From 383bdca0b4d20ad07527ee6c69f2aad1739778de Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 25 Mar 2026 16:44:32 -0400 Subject: [PATCH 01/22] DataFusion 53 Release Blog --- content/blog/2026-03-25-datafusion-53.0.0.md | 337 +++++++++++++++++++ 1 file changed, 337 insertions(+) create mode 100644 content/blog/2026-03-25-datafusion-53.0.0.md diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md new file mode 100644 index 00000000..66a1f72c --- /dev/null +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -0,0 +1,337 @@ +--- +layout: post +title: Apache DataFusion 53.0.0 Released +date: 2026-03-25 +author: pmc +categories: [release] +--- + + + +[TOC] + +We are proud to announce the release of [DataFusion 53.0.0]. This post highlights +some of the major improvements since [DataFusion 52.0.0]. The complete list of +changes is available in the [changelog]. Thanks to the [114 contributors] for +making this release possible. + +[DataFusion 53.0.0]: https://crates.io/crates/datafusion/53.0.0 +[DataFusion 52.0.0]: https://datafusion.apache.org/blog/2026/01/12/datafusion-52.0.0/ +[changelog]: https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md +[114 contributors]: https://github.com/apache/datafusion/blob/branch-53/dev/changelog/53.0.0.md#credits + +## Performance Improvements 🚀 + +DataFusion 53 continues the project-wide focus on reducing planning overhead, +skipping more unnecessary I/O, and pushing more work into earlier and cheaper +stages of execution. + +**TODO: Add performance chart and short discussion of the benchmark setup and results.** + +### `LIMIT`-Aware Parquet Row Group Pruning + +DataFusion 53 includes a new optimization that makes Parquet pruning aware of +SQL `LIMIT`. If DataFusion can prove that an entire row group matches the +predicate, and those fully matching row groups contain enough rows to satisfy +the `LIMIT`, it can skip scanning partially matching row groups entirely. + +This is important because it turns `LIMIT` from a result-shaping operator into +an I/O optimization. For selective queries on Parquet, this can avoid decoding +entire row groups that would previously have been scanned only to discover that +their rows were not needed. + +For example, consider: + +```sql +SELECT * +FROM tracking_data +WHERE species LIKE 'Alpine%' AND score >= 50 +LIMIT 3; +``` + +Conceptually, DataFusion 53 can now do: + +```text +Before: + scan partially matching row groups + scan fully matching row groups + stop after 3 matching rows + +After: + identify fully matching row groups from statistics + if they already contain >= 3 rows, scan only those row groups +``` + +For a deeper explanation, see the dedicated [limit pruning blog post]. +Related PRs: [#18868] + +### Broader Filter Pushdown Across Joins and Unions + +DataFusion 53 also pushes filters through more plan shapes. In particular, +predicates can now be inferred and pushed through more join types, and filters +can be pushed through `UnionExec` without being redundantly re-applied. + +This matters because the cheapest rows are the rows you never read. Better +pushdown means fewer rows flow into joins, repartitions, and later operators, +which reduces CPU, memory, and I/O. + +For example: + +```sql +SELECT * +FROM t1 +LEFT JOIN t2 ON t1.k = t2.k +WHERE t1.k = 1; +``` + +Now DataFusion can often transform the physical plan so the filter is applied +to both sides before the join: + +```text +Before: + HashJoinExec + DataSourceExec(t1) + DataSourceExec(t2) + FilterExec(k = 1) + +After: + HashJoinExec + FilterExec(k = 1) -> DataSourceExec(t1) + FilterExec(k = 1) -> DataSourceExec(t2) +``` + +Related PRs: [#19918], [#20145], [#20192] + +### Faster Query Planning + +DataFusion 53 reduces planning overhead by caching `PlanProperties`, +reusing them when plan children are unchanged, and making immutable pieces of +execution plans cheaper to clone. + +This work is important for applications that generate many queries, repeatedly +optimize similar plans, or build plans programmatically. In those environments, +planning cost can become user-visible even when execution is fast. + +Conceptually: + +```text +Before: + rebuild plan node + recompute PlanProperties + clone large immutable vectors + +After: + reuse cached PlanProperties when possible + share immutable plan state with Arc +``` + +Related PRs: [#19792], [#19893] + +## Major Features ✨ + +### Nested Field Pushdown for Cheap Expressions + +DataFusion 53 adds a new optimizer pipeline for pushing cheap +"move-towards-leaf" expressions such as `get_field` closer to the data source. +This allows nested field access to be extracted earlier in the plan rather than +materializing wider values and trimming them later. + +This is especially important for nested data such as structs. Instead of reading +an entire struct column and then extracting one field near the top of the plan, +DataFusion can push the field extraction down so only the needed data is carried +through the pipeline. + +For example: + +```sql +SELECT id, s['label'] +FROM t +WHERE s['value'] > 150; +``` + +The optimizer can rewrite the plan roughly like this: + +```text +Before: + ProjectionExec: get_field(s, 'label') + FilterExec: get_field(s, 'value') > 150 + DataSourceExec: projection=[id, s] + +After: + ProjectionExec: __label + FilterExec: __value > 150 + DataSourceExec: projection=[id, get_field(s,'label'), get_field(s,'value')] +``` + +This work also lays the foundation for deeper nested-data optimizations in +future releases. + +**TODO:** Verify whether we want to describe this primarily as a planning / +projection-pushdown improvement or emphasize the nested Parquet pruning angle. + +Related PRs: [#20065], [#20117], [#20239] + +### JSON Array File Support + +Prior to DataFusion 53, JSON support focused on newline-delimited JSON (NDJSON). +DataFusion 53 can now read JSON array files such as `[{...}, {...}]` directly, +including streaming inputs from object stores. + +This matters because many real-world systems export JSON in array form rather +than NDJSON. With this release, those datasets can be queried directly without +pre-converting them into a different layout. + +Example: + +```sql +CREATE EXTERNAL TABLE events +STORED AS JSON +OPTIONS ('format.newline_delimited' 'false') +LOCATION 's3://bucket/events.json'; +``` + +Conceptually, DataFusion reads the array in a streaming fashion: + +```text +JSON array file/stream + -> streaming array-to-NDJSON conversion + -> Arrow JSON reader + -> RecordBatch +``` + +Related PRs: [#19924] + +### Better Support for Programmatic and Extension-Oriented Engines + +DataFusion 53 continues to improve the APIs used by downstream systems and +extensions. Notable changes include support for extension planning for +`TableScan`, new parser support for `JsonAccess`/`:` as a binary operator, and +additional FFI improvements and fixes on the release branch. + +This is important because DataFusion is used as a library and planner inside +many other systems. Making the planner more extensible and the parser more +customizable lowers the cost of building specialized SQL dialects and data +sources on top of DataFusion. + +Example use cases include: + +```sql +-- extension-specific table syntax / planning +SELECT * FROM my_custom_source(...); + +-- variant / JSON style operators handled by an extension planner +SELECT payload:'user_id' FROM events; +``` + +**TODO:** Decide how much of this section should stay in the main post versus +move to a shorter "notable improvements" section. + +Related PRs: [#20548], [#20717], [#20722] + +## Other Notable Improvements + +In addition to the major features above, DataFusion 53 includes several other +changes that may be worth calling out in the final version of this post: + +### More SQL Surface Area + +DataFusion 53 added support for set-comparison subqueries, null-aware anti join, +limited deletion, and a large batch of Spark-compatible functions. + +These improvements make DataFusion easier to adopt for engines and workloads +that expect broader SQL compatibility. + +Related PRs: [#19109], [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] + +### More Visibility Into Parquet Pruning + +DataFusion 53 can now report the number of matched Parquet pages in +`DataSourceExec`, making it easier to understand whether pruning is working as +expected. + +This is useful both for users tuning workloads and for contributors developing +new pruning optimizations. + +Related PRs: [#19977] + +### Continued Work on Execution Correctness and Stability + +The `branch-53` release process pulled in a substantial number of fixes for +correctness and stability, including fixes for spilling, hash join edge cases, +FFI behavior, filter serialization, parquet leaf index mapping, and a regression +in `count(*)` with right anti / mark joins. + +These are not as headline-friendly as new features, but they are highly +important for real deployments. + +Related PRs: [#20726], [#20791], [#20792], [#20882], [#20883], [#20884], [#20890], [#20891], [#20892], [#20895], [#20898], [#20903], [#20918], [#20932] + +## Upgrade Notes + +DataFusion 53 also includes several breaking changes, including updates to the +SQL parser, optimizer behavior, and some physical-plan APIs. Please see the +[upgrade guide] and [changelog] for the full details before upgrading. + +[upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading.html + +## Thank You + +Thank you to everyone in the DataFusion community who contributed code, +reviews, testing, bug reports, documentation, and release engineering work for +53.0.0. + +[limit pruning blog post]: https://datafusion.apache.org/blog/2026/03/20/limit-pruning/ + +[#18868]: https://github.com/apache/datafusion/pull/18868 +[#19109]: https://github.com/apache/datafusion/pull/19109 +[#19635]: https://github.com/apache/datafusion/pull/19635 +[#19699]: https://github.com/apache/datafusion/pull/19699 +[#19702]: https://github.com/apache/datafusion/pull/19702 +[#19792]: https://github.com/apache/datafusion/pull/19792 +[#19811]: https://github.com/apache/datafusion/pull/19811 +[#19829]: https://github.com/apache/datafusion/pull/19829 +[#19845]: https://github.com/apache/datafusion/pull/19845 +[#19893]: https://github.com/apache/datafusion/pull/19893 +[#19918]: https://github.com/apache/datafusion/pull/19918 +[#19924]: https://github.com/apache/datafusion/pull/19924 +[#19977]: https://github.com/apache/datafusion/pull/19977 +[#20065]: https://github.com/apache/datafusion/pull/20065 +[#20117]: https://github.com/apache/datafusion/pull/20117 +[#20137]: https://github.com/apache/datafusion/pull/20137 +[#20145]: https://github.com/apache/datafusion/pull/20145 +[#20192]: https://github.com/apache/datafusion/pull/20192 +[#20239]: https://github.com/apache/datafusion/pull/20239 +[#20548]: https://github.com/apache/datafusion/pull/20548 +[#20717]: https://github.com/apache/datafusion/pull/20717 +[#20722]: https://github.com/apache/datafusion/pull/20722 +[#20726]: https://github.com/apache/datafusion/pull/20726 +[#20791]: https://github.com/apache/datafusion/pull/20791 +[#20792]: https://github.com/apache/datafusion/pull/20792 +[#20882]: https://github.com/apache/datafusion/pull/20882 +[#20883]: https://github.com/apache/datafusion/pull/20883 +[#20884]: https://github.com/apache/datafusion/pull/20884 +[#20890]: https://github.com/apache/datafusion/pull/20890 +[#20891]: https://github.com/apache/datafusion/pull/20891 +[#20892]: https://github.com/apache/datafusion/pull/20892 +[#20895]: https://github.com/apache/datafusion/pull/20895 +[#20898]: https://github.com/apache/datafusion/pull/20898 +[#20903]: https://github.com/apache/datafusion/pull/20903 +[#20918]: https://github.com/apache/datafusion/pull/20918 +[#20932]: https://github.com/apache/datafusion/pull/20932 From ea2756d9ed95b30cb02a6d62d008fdd65f52cad2 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 25 Mar 2026 17:46:11 -0400 Subject: [PATCH 02/22] Updates / cleanup --- content/blog/2026-03-25-datafusion-53.0.0.md | 51 +++++++++----------- 1 file changed, 22 insertions(+), 29 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 66a1f72c..c68692ed 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -45,19 +45,20 @@ stages of execution. **TODO: Add performance chart and short discussion of the benchmark setup and results.** +**TODO: Also mention we updated the clickbench benchmark to use DataFusion 52 ** + + ### `LIMIT`-Aware Parquet Row Group Pruning DataFusion 53 includes a new optimization that makes Parquet pruning aware of -SQL `LIMIT`. If DataFusion can prove that an entire row group matches the -predicate, and those fully matching row groups contain enough rows to satisfy -the `LIMIT`, it can skip scanning partially matching row groups entirely. - -This is important because it turns `LIMIT` from a result-shaping operator into -an I/O optimization. For selective queries on Parquet, this can avoid decoding -entire row groups that would previously have been scanned only to discover that -their rows were not needed. +SQL `LIMIT`, which is described in full in [limit pruning blog post]. If +DataFusion can prove that an entire row group matches the predicate, and those +fully matching row groups contain enough rows to satisfy the `LIMIT`, partially +matching row groups are skipped entirely. or selective queries on Parquet, this +optimization can avoid fetching entire row groups that would previously have +been scanned only to discover that their rows were not needed. -For example, consider: +For example, consider this example from the [limit pruning blog post]: ```sql SELECT * @@ -66,27 +67,17 @@ WHERE species LIKE 'Alpine%' AND score >= 50 LIMIT 3; ``` -Conceptually, DataFusion 53 can now do: - -```text -Before: - scan partially matching row groups - scan fully matching row groups - stop after 3 matching rows - -After: - identify fully matching row groups from statistics - if they already contain >= 3 rows, scan only those row groups -``` +Conceptually, prior to DataFusion 53, the engine would scan both partially and +fully matching row groups and stop after 3 matching rows. In DataFusion 53, if +the engine can identify fully matching row groups from statistics that already +contain >= 3 rows, it scans only those row groups -For a deeper explanation, see the dedicated [limit pruning blog post]. Related PRs: [#18868] -### Broader Filter Pushdown Across Joins and Unions +### Improved Filter Pushdown -DataFusion 53 also pushes filters through more plan shapes. In particular, -predicates can now be inferred and pushed through more join types, and filters -can be pushed through `UnionExec` without being redundantly re-applied. +DataFusion 53 pushes filters through more join types, and through `UnionExec`, +as well as has expanded support for pushing down [dynamic filters]. This matters because the cheapest rows are the rows you never read. Better pushdown means fewer rows flow into joins, repartitions, and later operators, @@ -293,9 +284,10 @@ SQL parser, optimizer behavior, and some physical-plan APIs. Please see the ## Thank You -Thank you to everyone in the DataFusion community who contributed code, -reviews, testing, bug reports, documentation, and release engineering work for -53.0.0. +Thank you to everyone in the DataFusion community who contributed code, reviews, +testing, bug reports, documentation, and release engineering work for 53.0.0. +This release contains direct contributions from 114 different people, and we are +grateful for the time and effort that everyone put in to make it happen. [limit pruning blog post]: https://datafusion.apache.org/blog/2026/03/20/limit-pruning/ @@ -335,3 +327,4 @@ reviews, testing, bug reports, documentation, and release engineering work for [#20903]: https://github.com/apache/datafusion/pull/20903 [#20918]: https://github.com/apache/datafusion/pull/20918 [#20932]: https://github.com/apache/datafusion/pull/20932 +[dynamic filters]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters From d9d80f9d0bd30106a485f62028408e818a410fb5 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 25 Mar 2026 17:55:24 -0400 Subject: [PATCH 03/22] more shouts --- content/blog/2026-03-25-datafusion-53.0.0.md | 152 +++++++++++++++++++ 1 file changed, 152 insertions(+) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index c68692ed..f6a75389 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -135,6 +135,30 @@ After: Related PRs: [#19792], [#19893] +### Faster Built-in Function Implementations + +DataFusion 53 also includes a large amount of low-level work to make built-in +scalar and array functions faster. This release contains optimizations for +string processing, array operations, and other frequently used expressions that +show up in real analytical workloads. + +This work is important because these functions sit on hot execution paths. +Small improvements to heavily used functions compound across large datasets and +complex queries. + +We especially want to call out [@neilconway] for a substantial series of +performance improvements in this release, including [strpos], [replace], +[concat], [translate], [array_has], [lpad]/[rpad], [trim], [initcap], +[to_array_of_size], [array_has_any], [array_position], [array_agg], and +[array_to_string]. + +Many other contributors also improved function performance in 53.0.0, including +[@theirix] on [left] and [right], [@lyne7-sc] on [array_remove], +[array_repeat_func], [array_union] / [array_intersect], and [array_distinct], +[@kumarUjjawal] on [ascii], [chr], [regexp_like], [atan2], [round], [signum], +and many other scalar fast paths, [@pepijnve] on [case_when], [@zhangxffff] on +[in_list], and [@UBarney] on [hash_table_lookup]. + ## Major Features ✨ ### Nested Field Pushdown for Cheap Expressions @@ -251,6 +275,23 @@ that expect broader SQL compatibility. Related PRs: [#19109], [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] +### More Spark Compatibility Functions + +DataFusion 53 added another substantial batch of Spark-compatible built-in +functions and behaviors, continuing the work to make DataFusion easier to use +for Spark-oriented workloads and downstream engines. + +This release includes new or improved support for functions such as + [collect_list] / [collect_set], [array_repeat_spark], [slice], [substring], + [date_diff], [add_months], [base64] / [unbase64], [from_utc_timestamp] / + [to_utc_timestamp], [unix_date] and [unix_timestamp], [negative], + [string_to_map], [json_tuple], [arrays_zip], [bitmap_bit_position], + [bitmap_bucket_number], [bin], [abs_ansi], and [array_contains]. + +Special thanks to the contributors who drove this work, especially [@cht42], +[@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], +[@aryan-212], [@hsiang-c], and [@davidlghellin]. + ### More Visibility Into Parquet Pruning DataFusion 53 can now report the number of matched Parquet pages in @@ -290,27 +331,138 @@ This release contains direct contributions from 114 different people, and we are grateful for the time and effort that everyone put in to make it happen. [limit pruning blog post]: https://datafusion.apache.org/blog/2026/03/20/limit-pruning/ +[@neilconway]: https://github.com/neilconway +[@cht42]: https://github.com/cht42 +[@CuteChuanChuan]: https://github.com/CuteChuanChuan +[@SubhamSinghal]: https://github.com/SubhamSinghal +[@kazantsev-maksim]: https://github.com/kazantsev-maksim +[@unknowntpo]: https://github.com/unknowntpo +[@aryan-212]: https://github.com/aryan-212 +[@hsiang-c]: https://github.com/hsiang-c +[@davidlghellin]: https://github.com/davidlghellin +[@theirix]: https://github.com/theirix +[@lyne7-sc]: https://github.com/lyne7-sc +[@kumarUjjawal]: https://github.com/kumarUjjawal +[@pepijnve]: https://github.com/pepijnve +[@zhangxffff]: https://github.com/zhangxffff +[@UBarney]: https://github.com/UBarney + +[abs_ansi]: https://github.com/apache/datafusion/pull/18828 +[add_months]: https://github.com/apache/datafusion/pull/19711 +[array_agg]: https://github.com/apache/datafusion/pull/20504 +[array_contains]: https://github.com/apache/datafusion/pull/20685 +[array_distinct]: https://github.com/apache/datafusion/pull/20364 +[array_has]: https://github.com/apache/datafusion/pull/20374 +[array_has_any]: https://github.com/apache/datafusion/pull/20385 +[array_intersect]: https://github.com/apache/datafusion/pull/20243 +[array_position]: https://github.com/apache/datafusion/pull/20532 +[array_remove]: https://github.com/apache/datafusion/pull/19996 +[array_repeat_func]: https://github.com/apache/datafusion/pull/20049 +[array_repeat_spark]: https://github.com/apache/datafusion/pull/19702 +[array_to_string]: https://github.com/apache/datafusion/pull/20553 +[array_union]: https://github.com/apache/datafusion/pull/20243 +[arrays_zip]: https://github.com/apache/datafusion/pull/20440 +[ascii]: https://github.com/apache/datafusion/pull/19951 +[atan2]: https://github.com/apache/datafusion/pull/20336 +[base64]: https://github.com/apache/datafusion/pull/19968 +[bin]: https://github.com/apache/datafusion/pull/20479 +[bitmap_bit_position]: https://github.com/apache/datafusion/pull/20275 +[bitmap_bucket_number]: https://github.com/apache/datafusion/pull/20288 +[case_when]: https://github.com/apache/datafusion/pull/20097 +[chr]: https://github.com/apache/datafusion/pull/20073 +[collect_list]: https://github.com/apache/datafusion/pull/19699 +[collect_set]: https://github.com/apache/datafusion/pull/19699 +[concat]: https://github.com/apache/datafusion/pull/20317 +[date_diff]: https://github.com/apache/datafusion/pull/19845 +[from_utc_timestamp]: https://github.com/apache/datafusion/pull/19880 +[hash_table_lookup]: https://github.com/apache/datafusion/pull/19602 +[in_list]: https://github.com/apache/datafusion/pull/20528 +[initcap]: https://github.com/apache/datafusion/pull/20352 +[json_tuple]: https://github.com/apache/datafusion/pull/20412 +[left]: https://github.com/apache/datafusion/pull/19980 +[lpad]: https://github.com/apache/datafusion/pull/20278 +[negative]: https://github.com/apache/datafusion/pull/20006 +[regexp_like]: https://github.com/apache/datafusion/pull/20354 +[replace]: https://github.com/apache/datafusion/pull/20344 +[right]: https://github.com/apache/datafusion/pull/20069 +[round]: https://github.com/apache/datafusion/pull/19831 +[rpad]: https://github.com/apache/datafusion/pull/20278 +[signum]: https://github.com/apache/datafusion/pull/19871 +[slice]: https://github.com/apache/datafusion/pull/19811 +[strpos]: https://github.com/apache/datafusion/pull/20295 +[string_to_map]: https://github.com/apache/datafusion/pull/20120 +[substring]: https://github.com/apache/datafusion/pull/19805 +[to_array_of_size]: https://github.com/apache/datafusion/pull/20459 +[to_utc_timestamp]: https://github.com/apache/datafusion/pull/19880 +[translate]: https://github.com/apache/datafusion/pull/20305 +[trim]: https://github.com/apache/datafusion/pull/20328 +[unbase64]: https://github.com/apache/datafusion/pull/19968 +[unix_date]: https://github.com/apache/datafusion/pull/19892 +[unix_timestamp]: https://github.com/apache/datafusion/pull/19892 [#18868]: https://github.com/apache/datafusion/pull/18868 +[#18828]: https://github.com/apache/datafusion/pull/18828 [#19109]: https://github.com/apache/datafusion/pull/19109 +[#19592]: https://github.com/apache/datafusion/pull/19592 [#19635]: https://github.com/apache/datafusion/pull/19635 [#19699]: https://github.com/apache/datafusion/pull/19699 [#19702]: https://github.com/apache/datafusion/pull/19702 +[#19711]: https://github.com/apache/datafusion/pull/19711 [#19792]: https://github.com/apache/datafusion/pull/19792 +[#19805]: https://github.com/apache/datafusion/pull/19805 [#19811]: https://github.com/apache/datafusion/pull/19811 [#19829]: https://github.com/apache/datafusion/pull/19829 [#19845]: https://github.com/apache/datafusion/pull/19845 +[#19865]: https://github.com/apache/datafusion/pull/19865 +[#19880]: https://github.com/apache/datafusion/pull/19880 +[#19892]: https://github.com/apache/datafusion/pull/19892 [#19893]: https://github.com/apache/datafusion/pull/19893 [#19918]: https://github.com/apache/datafusion/pull/19918 [#19924]: https://github.com/apache/datafusion/pull/19924 +[#19951]: https://github.com/apache/datafusion/pull/19951 +[#19996]: https://github.com/apache/datafusion/pull/19996 +[#19968]: https://github.com/apache/datafusion/pull/19968 +[#19984]: https://github.com/apache/datafusion/pull/19984 [#19977]: https://github.com/apache/datafusion/pull/19977 +[#20049]: https://github.com/apache/datafusion/pull/20049 [#20065]: https://github.com/apache/datafusion/pull/20065 +[#20006]: https://github.com/apache/datafusion/pull/20006 +[#20073]: https://github.com/apache/datafusion/pull/20073 +[#20097]: https://github.com/apache/datafusion/pull/20097 [#20117]: https://github.com/apache/datafusion/pull/20117 [#20137]: https://github.com/apache/datafusion/pull/20137 [#20145]: https://github.com/apache/datafusion/pull/20145 [#20192]: https://github.com/apache/datafusion/pull/20192 +[#20120]: https://github.com/apache/datafusion/pull/20120 +[#20243]: https://github.com/apache/datafusion/pull/20243 [#20239]: https://github.com/apache/datafusion/pull/20239 +[#20275]: https://github.com/apache/datafusion/pull/20275 +[#20288]: https://github.com/apache/datafusion/pull/20288 +[#20278]: https://github.com/apache/datafusion/pull/20278 +[#20295]: https://github.com/apache/datafusion/pull/20295 +[#20305]: https://github.com/apache/datafusion/pull/20305 +[#20317]: https://github.com/apache/datafusion/pull/20317 +[#20323]: https://github.com/apache/datafusion/pull/20323 +[#20328]: https://github.com/apache/datafusion/pull/20328 +[#20336]: https://github.com/apache/datafusion/pull/20336 +[#20344]: https://github.com/apache/datafusion/pull/20344 +[#20354]: https://github.com/apache/datafusion/pull/20354 +[#20352]: https://github.com/apache/datafusion/pull/20352 +[#20374]: https://github.com/apache/datafusion/pull/20374 +[#20385]: https://github.com/apache/datafusion/pull/20385 +[#20364]: https://github.com/apache/datafusion/pull/20364 +[#20412]: https://github.com/apache/datafusion/pull/20412 +[#20440]: https://github.com/apache/datafusion/pull/20440 +[#20461]: https://github.com/apache/datafusion/pull/20461 +[#20479]: https://github.com/apache/datafusion/pull/20479 +[#20459]: https://github.com/apache/datafusion/pull/20459 [#20548]: https://github.com/apache/datafusion/pull/20548 +[#20504]: https://github.com/apache/datafusion/pull/20504 +[#20532]: https://github.com/apache/datafusion/pull/20532 +[#20538]: https://github.com/apache/datafusion/pull/20538 +[#20553]: https://github.com/apache/datafusion/pull/20553 +[#20528]: https://github.com/apache/datafusion/pull/20528 +[#20685]: https://github.com/apache/datafusion/pull/20685 [#20717]: https://github.com/apache/datafusion/pull/20717 [#20722]: https://github.com/apache/datafusion/pull/20722 [#20726]: https://github.com/apache/datafusion/pull/20726 From c4691f3eaf424d1af9e32beba4b7005c864500fc Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 25 Mar 2026 17:58:01 -0400 Subject: [PATCH 04/22] clenaup --- content/blog/2026-03-25-datafusion-53.0.0.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index f6a75389..02cd65cf 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -281,12 +281,9 @@ DataFusion 53 added another substantial batch of Spark-compatible built-in functions and behaviors, continuing the work to make DataFusion easier to use for Spark-oriented workloads and downstream engines. -This release includes new or improved support for functions such as - [collect_list] / [collect_set], [array_repeat_spark], [slice], [substring], - [date_diff], [add_months], [base64] / [unbase64], [from_utc_timestamp] / - [to_utc_timestamp], [unix_date] and [unix_timestamp], [negative], - [string_to_map], [json_tuple], [arrays_zip], [bitmap_bit_position], - [bitmap_bucket_number], [bin], [abs_ansi], and [array_contains]. +This release includes around 20 new or improved Spark-compatible functions and +behaviors, such as [collect_list], [date_diff], [from_utc_timestamp], +[json_tuple], [arrays_zip], [bin], and [array_contains]. Special thanks to the contributors who drove this work, especially [@cht42], [@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], From 52725fad99c577d0d8a4361e2dd12a8bf0ca556d Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 08:35:11 -0400 Subject: [PATCH 05/22] Edits --- content/blog/2026-03-25-datafusion-53.0.0.md | 107 ++++++++++-------- .../datafusion-53/join-filter-pushdown.svg | 68 +++++++++++ 2 files changed, 130 insertions(+), 45 deletions(-) create mode 100644 content/images/datafusion-53/join-filter-pushdown.svg diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 02cd65cf..ecb231c1 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -39,9 +39,9 @@ making this release possible. ## Performance Improvements 🚀 -DataFusion 53 continues the project-wide focus on reducing planning overhead, -skipping more unnecessary I/O, and pushing more work into earlier and cheaper -stages of execution. +DataFusion 53 continues the project-wide focus on performance in general and +this release features reducing planning overhead, skipping more unnecessary I/O, +and pushing more work into earlier and cheaper stages of execution. **TODO: Add performance chart and short discussion of the benchmark setup and results.** @@ -54,32 +54,19 @@ DataFusion 53 includes a new optimization that makes Parquet pruning aware of SQL `LIMIT`, which is described in full in [limit pruning blog post]. If DataFusion can prove that an entire row group matches the predicate, and those fully matching row groups contain enough rows to satisfy the `LIMIT`, partially -matching row groups are skipped entirely. or selective queries on Parquet, this -optimization can avoid fetching entire row groups that would previously have -been scanned only to discover that their rows were not needed. +matching row groups are skipped entirely. -For example, consider this example from the [limit pruning blog post]: +
+Pruning pipeline with limit pruning highlighted +
Figure 1: New limit pruning inserted between row group and page index pruning.
+
-```sql -SELECT * -FROM tracking_data -WHERE species LIKE 'Alpine%' AND score >= 50 -LIMIT 3; -``` - -Conceptually, prior to DataFusion 53, the engine would scan both partially and -fully matching row groups and stop after 3 matching rows. In DataFusion 53, if -the engine can identify fully matching row groups from statistics that already -contain >= 3 rows, it scans only those row groups - -Related PRs: [#18868] +Special thanks to [@xudong963] for implementing this feature. Related PRs: [#18868] ### Improved Filter Pushdown -DataFusion 53 pushes filters through more join types, and through `UnionExec`, -as well as has expanded support for pushing down [dynamic filters]. - -This matters because the cheapest rows are the rows you never read. Better +DataFusion 53 pushes filters down through more join types, and through `UnionExec`, +as well as has expanded support for pushing down [dynamic filters]. More pushdown means fewer rows flow into joins, repartitions, and later operators, which reduces CPU, memory, and I/O. @@ -108,36 +95,30 @@ After: FilterExec(k = 1) -> DataSourceExec(t2) ``` -Related PRs: [#19918], [#20145], [#20192] +
+Before and after diagram of filter pushdown around a hash join +
Figure 2: DataFusion 53 infers additional filters are from join conditions and pushes them down in the plan
+
-### Faster Query Planning - -DataFusion 53 reduces planning overhead by caching `PlanProperties`, -reusing them when plan children are unchanged, and making immutable pieces of -execution plans cheaper to clone. +Special thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for +driving this work. Related PRs: [#19918], [#20145], [#20192] -This work is important for applications that generate many queries, repeatedly -optimize similar plans, or build plans programmatically. In those environments, -planning cost can become user-visible even when execution is fast. +### Faster Query Planning -Conceptually: +DataFusion 53 improves query planning performance, making immutable pieces of +execution plans cheaper to clone. This helps applications that need extreme low +latency, plan many or complex queries, and those using prepared statements or +parameterized queries. -```text -Before: - rebuild plan node - recompute PlanProperties - clone large immutable vectors - -After: - reuse cached PlanProperties when possible - share immutable plan state with Arc -``` +Special thanks to [@askalt] for leading this work. planning overhead by caching `PlanProperties`, Related PRs: [#19792], [#19893] +** TODO get sql_planner benchmarks for 52.4.0 and 53.0.0 ** + ### Faster Built-in Function Implementations -DataFusion 53 also includes a large amount of low-level work to make built-in +DataFusion 53 also includes a significant number of low-level optimizations to built-in scalar and array functions faster. This release contains optimizations for string processing, array operations, and other frequently used expressions that show up in real analytical workloads. @@ -201,6 +182,9 @@ future releases. **TODO:** Verify whether we want to describe this primarily as a planning / projection-pushdown improvement or emphasize the nested Parquet pruning angle. +Special thanks to [@adriangb] for designing and implementing this optimizer +work. + Related PRs: [#20065], [#20117], [#20239] ### JSON Array File Support @@ -231,6 +215,8 @@ JSON array file/stream -> RecordBatch ``` +Special thanks to [@zhuqi-lucas] for implementing this feature. + Related PRs: [#19924] ### Better Support for Programmatic and Extension-Oriented Engines @@ -258,6 +244,9 @@ SELECT payload:'user_id' FROM events; **TODO:** Decide how much of this section should stay in the main post versus move to a shorter "notable improvements" section. +Special thanks to [@linhr], [@Samyak2], and [@timsaucer] for these +extension-oriented improvements. + Related PRs: [#20548], [#20717], [#20722] ## Other Notable Improvements @@ -273,6 +262,9 @@ limited deletion, and a large batch of Spark-compatible functions. These improvements make DataFusion easier to adopt for engines and workloads that expect broader SQL compatibility. +Special thanks to [@waynexia], [@viirya], [@askalt], and [@cht42] for key +contributions in this area. + Related PRs: [#19109], [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] ### More Spark Compatibility Functions @@ -298,6 +290,8 @@ expected. This is useful both for users tuning workloads and for contributors developing new pruning optimizations. +Special thanks to [@nuno-faria] for this visibility improvement. + Related PRs: [#19977] ### Continued Work on Execution Correctness and Stability @@ -310,6 +304,10 @@ in `count(*)` with right anti / mark joins. These are not as headline-friendly as new features, but they are highly important for real deployments. +Thanks to [@jonathanc-n], [@alamb], [@xanderbailey], [@haohuaijin], +[@friendlymatthew], [@fwojciec], [@Kontinuation], [@nathanb9], and many others +who helped stabilize the release branch. + Related PRs: [#20726], [#20791], [#20792], [#20882], [#20883], [#20884], [#20890], [#20891], [#20892], [#20895], [#20898], [#20903], [#20918], [#20932] ## Upgrade Notes @@ -329,6 +327,25 @@ grateful for the time and effort that everyone put in to make it happen. [limit pruning blog post]: https://datafusion.apache.org/blog/2026/03/20/limit-pruning/ [@neilconway]: https://github.com/neilconway +[@xudong963]: https://github.com/xudong963 +[@nuno-faria]: https://github.com/nuno-faria +[@haohuaijin]: https://github.com/haohuaijin +[@jackkleeman]: https://github.com/jackkleeman +[@askalt]: https://github.com/askalt +[@adriangb]: https://github.com/adriangb +[@zhuqi-lucas]: https://github.com/zhuqi-lucas +[@linhr]: https://github.com/linhr +[@Samyak2]: https://github.com/Samyak2 +[@timsaucer]: https://github.com/timsaucer +[@waynexia]: https://github.com/waynexia +[@viirya]: https://github.com/viirya +[@jonathanc-n]: https://github.com/jonathanc-n +[@alamb]: https://github.com/alamb +[@xanderbailey]: https://github.com/xanderbailey +[@friendlymatthew]: https://github.com/friendlymatthew +[@fwojciec]: https://github.com/fwojciec +[@Kontinuation]: https://github.com/Kontinuation +[@nathanb9]: https://github.com/nathanb9 [@cht42]: https://github.com/cht42 [@CuteChuanChuan]: https://github.com/CuteChuanChuan [@SubhamSinghal]: https://github.com/SubhamSinghal diff --git a/content/images/datafusion-53/join-filter-pushdown.svg b/content/images/datafusion-53/join-filter-pushdown.svg new file mode 100644 index 00000000..84a51472 --- /dev/null +++ b/content/images/datafusion-53/join-filter-pushdown.svg @@ -0,0 +1,68 @@ + + + + + + + + + + Before + + + + FilterExec + k = 1 + + + + + HashJoinExec + + + + + + DataSourceExec + t1 + + + DataSourceExec + t2 + + + After + + + + HashJoinExec + + + + + + FilterExec + k = 1 + + + FilterExec + k = 1 + + + + + + DataSourceExec + t1 + + + DataSourceExec + t2 + From c03f0e0a1d22aad4e643d9410c80ea6468d9deb4 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 08:45:13 -0400 Subject: [PATCH 06/22] update --- content/blog/2026-03-25-datafusion-53.0.0.md | 32 +++++++++----------- 1 file changed, 14 insertions(+), 18 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index ecb231c1..55bfdb86 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -118,23 +118,18 @@ Related PRs: [#19792], [#19893] ### Faster Built-in Function Implementations -DataFusion 53 also includes a significant number of low-level optimizations to built-in -scalar and array functions faster. This release contains optimizations for -string processing, array operations, and other frequently used expressions that -show up in real analytical workloads. - -This work is important because these functions sit on hot execution paths. -Small improvements to heavily used functions compound across large datasets and -complex queries. - -We especially want to call out [@neilconway] for a substantial series of -performance improvements in this release, including [strpos], [replace], -[concat], [translate], [array_has], [lpad]/[rpad], [trim], [initcap], -[to_array_of_size], [array_has_any], [array_position], [array_agg], and -[array_to_string]. - -Many other contributors also improved function performance in 53.0.0, including -[@theirix] on [left] and [right], [@lyne7-sc] on [array_remove], +DataFusion 53 includes performance improvements for many built-in functions. +DataFusion, like all real-world SQL engines, includes [235 built-in +functions], and improving the performance of these functions benefits a wide +range of workloads. + + +Special thanks to [@neilconway] for a substantial series of performance +improvements in this release, including [strpos], [replace], [concat], +[translate], [array_has], [lpad]/[rpad], [trim], [initcap], [to_array_of_size], +[array_has_any], [array_position], [array_agg], and [array_to_string]. In +addition, other contributors also improved function performance in 53.0.0, +including [@theirix] on [left] and [right], [@lyne7-sc] on [array_remove], [array_repeat_func], [array_union] / [array_intersect], and [array_distinct], [@kumarUjjawal] on [ascii], [chr], [regexp_like], [atan2], [round], [signum], and many other scalar fast paths, [@pepijnve] on [case_when], [@zhangxffff] on @@ -273,7 +268,7 @@ DataFusion 53 added another substantial batch of Spark-compatible built-in functions and behaviors, continuing the work to make DataFusion easier to use for Spark-oriented workloads and downstream engines. -This release includes around 20 new or improved Spark-compatible functions and +This release includes almost 20 new or improved Spark-compatible functions and behaviors, such as [collect_list], [date_diff], [from_utc_timestamp], [json_tuple], [arrays_zip], [bin], and [array_contains]. @@ -494,3 +489,4 @@ grateful for the time and effort that everyone put in to make it happen. [#20918]: https://github.com/apache/datafusion/pull/20918 [#20932]: https://github.com/apache/datafusion/pull/20932 [dynamic filters]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters +[235 built-in functions]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html \ No newline at end of file From 2f5a6f24f81b68437b516b10fdf7a62aeedb9ada Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 08:58:31 -0400 Subject: [PATCH 07/22] update --- content/blog/2026-03-25-datafusion-53.0.0.md | 91 ++++++------------- .../datafusion-53/field-access-pushdown.svg | 58 ++++++++++++ 2 files changed, 84 insertions(+), 65 deletions(-) create mode 100644 content/images/datafusion-53/field-access-pushdown.svg diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 55bfdb86..02d8a4ea 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -123,33 +123,25 @@ DataFusion, like all real-world SQL engines, includes [235 built-in functions], and improving the performance of these functions benefits a wide range of workloads. +This release includes performance improvements for 42 distinct built-in +functions, such as [strpos], [replace], [concat], [translate], [array_has], +[array_agg], [left], [right], and [case_when]. -Special thanks to [@neilconway] for a substantial series of performance -improvements in this release, including [strpos], [replace], [concat], -[translate], [array_has], [lpad]/[rpad], [trim], [initcap], [to_array_of_size], -[array_has_any], [array_position], [array_agg], and [array_to_string]. In -addition, other contributors also improved function performance in 53.0.0, -including [@theirix] on [left] and [right], [@lyne7-sc] on [array_remove], -[array_repeat_func], [array_union] / [array_intersect], and [array_distinct], -[@kumarUjjawal] on [ascii], [chr], [regexp_like], [atan2], [round], [signum], -and many other scalar fast paths, [@pepijnve] on [case_when], [@zhangxffff] on -[in_list], and [@UBarney] on [hash_table_lookup]. +Special thanks to the contributors who drove this work, especially +[@neilconway], [@theirix], [@lyne7-sc], [@kumarUjjawal], [@pepijnve], +[@zhangxffff], and [@UBarney]. -## Major Features ✨ -### Nested Field Pushdown for Cheap Expressions +### Nested Field Pushdown -DataFusion 53 adds a new optimizer pipeline for pushing cheap -"move-towards-leaf" expressions such as `get_field` closer to the data source. -This allows nested field access to be extracted earlier in the plan rather than -materializing wider values and trimming them later. +DataFusion 53 can now push cheap expressions such as `get_field` down the plan +and into data sources. This is especially important for nested data such as +structs in Parquet files. Instead of reading an entire struct column and then +extracting the field of interest, DataFusion 53 pushes the field extraction into +the scan.. -This is especially important for nested data such as structs. Instead of reading -an entire struct column and then extracting one field near the top of the plan, -DataFusion can push the field extraction down so only the needed data is carried -through the pipeline. - -For example: +For example the following query reads a struct column `s` and extracts the +`label` field, for rows where the `value` field is greater than 150: ```sql SELECT id, s['label'] @@ -171,53 +163,22 @@ After: DataSourceExec: projection=[id, get_field(s,'label'), get_field(s,'value')] ``` -This work also lays the foundation for deeper nested-data optimizations in -future releases. - -**TODO:** Verify whether we want to describe this primarily as a planning / -projection-pushdown improvement or emphasize the nested Parquet pruning angle. +
+Before and after diagram of field access pushdown into a data source +
Figure 3: DataFusion 53 pushes field-access expressions closer to the scan.
+
Special thanks to [@adriangb] for designing and implementing this optimizer -work. - -Related PRs: [#20065], [#20117], [#20239] - -### JSON Array File Support - -Prior to DataFusion 53, JSON support focused on newline-delimited JSON (NDJSON). -DataFusion 53 can now read JSON array files such as `[{...}, {...}]` directly, -including streaming inputs from object stores. - -This matters because many real-world systems export JSON in array form rather -than NDJSON. With this release, those datasets can be queried directly without -pre-converting them into a different layout. - -Example: - -```sql -CREATE EXTERNAL TABLE events -STORED AS JSON -OPTIONS ('format.newline_delimited' 'false') -LOCATION 's3://bucket/events.json'; -``` - -Conceptually, DataFusion reads the array in a streaming fashion: - -```text -JSON array file/stream - -> streaming array-to-NDJSON conversion - -> Arrow JSON reader - -> RecordBatch -``` - -Special thanks to [@zhuqi-lucas] for implementing this feature. +work. Related PRs: [#20065], [#20117], [#20239] -Related PRs: [#19924] +## New Features ✨ -### Better Support for Programmatic and Extension-Oriented Engines +* **JSON Array File Support**: DataFusion 53 can now read JSON arrays such as + `[{...}, {...}]` directly as multiple rows, including streaming inputs from + object stores. Special thanks to [@zhuqi-lucas] for implementing this + feature. Related PRs: [#19924] -DataFusion 53 continues to improve the APIs used by downstream systems and -extensions. Notable changes include support for extension planning for +* **Variant extension planning for `TableScan`, new parser support for `JsonAccess`/`:` as a binary operator, and additional FFI improvements and fixes on the release branch. @@ -489,4 +450,4 @@ grateful for the time and effort that everyone put in to make it happen. [#20918]: https://github.com/apache/datafusion/pull/20918 [#20932]: https://github.com/apache/datafusion/pull/20932 [dynamic filters]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters -[235 built-in functions]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html \ No newline at end of file +[235 built-in functions]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html diff --git a/content/images/datafusion-53/field-access-pushdown.svg b/content/images/datafusion-53/field-access-pushdown.svg new file mode 100644 index 00000000..e0c7b90f --- /dev/null +++ b/content/images/datafusion-53/field-access-pushdown.svg @@ -0,0 +1,58 @@ + + + + + + + + + + Before + + + + ProjectionExec + s['label'] + + + + + FilterExec + s['value'] > 150 + + + + + DataSourceExec + projection=[id, s] + reads full struct column + + + After + + + + ProjectionExec + __label + + + + + FilterExec + __value > 150 + + + + + DataSourceExec + projection=[id, s['label'], s['value']] + field access extracted near the scan + From 1aeed35cdaad2fb12333d0daf7b50197a394d8dc Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 10:19:39 -0400 Subject: [PATCH 08/22] figure --- .../datafusion-53/join-filter-pushdown.svg | 56 +++++++++---------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/content/images/datafusion-53/join-filter-pushdown.svg b/content/images/datafusion-53/join-filter-pushdown.svg index 84a51472..68bd1101 100644 --- a/content/images/datafusion-53/join-filter-pushdown.svg +++ b/content/images/datafusion-53/join-filter-pushdown.svg @@ -6,63 +6,63 @@ Before - - FilterExec - k = 1 + + FilterExec + k = 1 - + - HashJoinExec + HashJoinExec - + DataSourceExec - t1 + t1 - + DataSourceExec - t2 + t2 After - - HashJoinExec + + HashJoinExec - - + + - - FilterExec - k = 1 + + FilterExec + k = 1 - - FilterExec - k = 1 + + FilterExec + k = 1 - - + + - + DataSourceExec - t1 + t1 - + DataSourceExec - t2 + t2 From 3935a305bb3355d8b454f265dabd473564b81f2e Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 10:27:57 -0400 Subject: [PATCH 09/22] More updates --- content/blog/2026-03-25-datafusion-53.0.0.md | 81 ++++---------------- 1 file changed, 16 insertions(+), 65 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 02d8a4ea..8697d13e 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -1,4 +1,4 @@ ---- +layout--- layout: post title: Apache DataFusion 53.0.0 Released date: 2026-03-25 @@ -149,20 +149,6 @@ FROM t WHERE s['value'] > 150; ``` -The optimizer can rewrite the plan roughly like this: - -```text -Before: - ProjectionExec: get_field(s, 'label') - FilterExec: get_field(s, 'value') > 150 - DataSourceExec: projection=[id, s] - -After: - ProjectionExec: __label - FilterExec: __value > 150 - DataSourceExec: projection=[id, get_field(s,'label'), get_field(s,'value')] -``` -
Before and after diagram of field access pushdown into a data source
Figure 3: DataFusion 53 pushes field-access expressions closer to the scan.
@@ -178,60 +164,23 @@ work. Related PRs: [#20065], [#20117], [#20239] object stores. Special thanks to [@zhuqi-lucas] for implementing this feature. Related PRs: [#19924] -* **Variant extension planning for -`TableScan`, new parser support for `JsonAccess`/`:` as a binary operator, and -additional FFI improvements and fixes on the release branch. - -This is important because DataFusion is used as a library and planner inside -many other systems. Making the planner more extensible and the parser more -customizable lowers the cost of building specialized SQL dialects and data -sources on top of DataFusion. - -Example use cases include: - -```sql --- extension-specific table syntax / planning -SELECT * FROM my_custom_source(...); - --- variant / JSON style operators handled by an extension planner -SELECT payload:'user_id' FROM events; -``` - -**TODO:** Decide how much of this section should stay in the main post versus -move to a shorter "notable improvements" section. - -Special thanks to [@linhr], [@Samyak2], and [@timsaucer] for these -extension-oriented improvements. - -Related PRs: [#20548], [#20717], [#20722] - -## Other Notable Improvements - -In addition to the major features above, DataFusion 53 includes several other -changes that may be worth calling out in the final version of this post: - -### More SQL Surface Area - -DataFusion 53 added support for set-comparison subqueries, null-aware anti join, -limited deletion, and a large batch of Spark-compatible functions. - -These improvements make DataFusion easier to adopt for engines and workloads -that expect broader SQL compatibility. - -Special thanks to [@waynexia], [@viirya], [@askalt], and [@cht42] for key -contributions in this area. +* **Support for `:` operator**: DataFusion can plan queries such as + `SELECT payload:'user_id' FROM events;`, enabling better [Parquet Variant] + support via [datafusion-variant]. Thanks to [@Samyak2], -Related PRs: [#19109], [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] -### More Spark Compatibility Functions +* SQL**: DataFusion supports additional set-comparison subqueries, null-aware + anti join, deletion predicates. Thanks to [@waynexia], [@viirya], [@askalt], + and [@cht42] for key contributions in this area. Related PRs: [#19109], + [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] -DataFusion 53 added another substantial batch of Spark-compatible built-in -functions and behaviors, continuing the work to make DataFusion easier to use -for Spark-oriented workloads and downstream engines. +### More Spark Functions -This release includes almost 20 new or improved Spark-compatible functions and -behaviors, such as [collect_list], [date_diff], [from_utc_timestamp], -[json_tuple], [arrays_zip], [bin], and [array_contains]. +DataFusion 53 includes almost 20 new or improved Spark-compatible functions and +behaviors in the [`datafusion-spark` crate], making it easier to use for +Spark-oriented workloads and downstream engines. This release contains functions +such as [collect_list], [date_diff], [from_utc_timestamp], [json_tuple], +[arrays_zip], [bin], and [array_contains]. Special thanks to the contributors who drove this work, especially [@cht42], [@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], @@ -451,3 +400,5 @@ grateful for the time and effort that everyone put in to make it happen. [#20932]: https://github.com/apache/datafusion/pull/20932 [dynamic filters]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters [235 built-in functions]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html +[datafusion-variant]: https://github.com/datafusion-contrib/datafusion-variant +[Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ \ No newline at end of file From c7b0478604d28dc1db364a92d6ffb69e790b2184 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 10:40:30 -0400 Subject: [PATCH 10/22] updates --- content/blog/2026-03-25-datafusion-53.0.0.md | 124 +++++++------------ 1 file changed, 44 insertions(+), 80 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 8697d13e..1270076a 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -1,4 +1,4 @@ -layout--- +--- layout: post title: Apache DataFusion 53.0.0 Released date: 2026-03-25 @@ -61,7 +61,7 @@ matching row groups are skipped entirely.
Figure 1: New limit pruning inserted between row group and page index pruning.
-Special thanks to [@xudong963] for implementing this feature. Related PRs: [#18868] +Thanks to [@xudong963] for implementing this feature. Related PRs: [#18868] ### Improved Filter Pushdown @@ -80,27 +80,14 @@ WHERE t1.k = 1; ``` Now DataFusion can often transform the physical plan so the filter is applied -to both sides before the join: - -```text -Before: - HashJoinExec - DataSourceExec(t1) - DataSourceExec(t2) - FilterExec(k = 1) - -After: - HashJoinExec - FilterExec(k = 1) -> DataSourceExec(t1) - FilterExec(k = 1) -> DataSourceExec(t2) -``` +to both sides before the join. This is especially important with [dynamic filters].
Before and after diagram of filter pushdown around a hash join
Figure 2: DataFusion 53 infers additional filters are from join conditions and pushes them down in the plan
-Special thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for +Thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for driving this work. Related PRs: [#19918], [#20145], [#20192] ### Faster Query Planning @@ -108,37 +95,31 @@ driving this work. Related PRs: [#19918], [#20145], [#20192] DataFusion 53 improves query planning performance, making immutable pieces of execution plans cheaper to clone. This helps applications that need extreme low latency, plan many or complex queries, and those using prepared statements or -parameterized queries. +parameterized queries. -Special thanks to [@askalt] for leading this work. planning overhead by caching `PlanProperties`, - -Related PRs: [#19792], [#19893] +Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] ** TODO get sql_planner benchmarks for 52.4.0 and 53.0.0 ** -### Faster Built-in Function Implementations - -DataFusion 53 includes performance improvements for many built-in functions. -DataFusion, like all real-world SQL engines, includes [235 built-in -functions], and improving the performance of these functions benefits a wide -range of workloads. +### Faster Functions -This release includes performance improvements for 42 distinct built-in +DataFusion, like all real-world SQL engines, includes [235 built-in functions], +and improving the performance of these functions benefits a wide range of +workloads. This release includes performance improvements for 42 of those functions, such as [strpos], [replace], [concat], [translate], [array_has], [array_agg], [left], [right], and [case_when]. -Special thanks to the contributors who drove this work, especially -[@neilconway], [@theirix], [@lyne7-sc], [@kumarUjjawal], [@pepijnve], -[@zhangxffff], and [@UBarney]. +Thanks to the contributors who drove this work, especially [@neilconway], +[@theirix], [@lyne7-sc], [@kumarUjjawal], [@pepijnve], [@zhangxffff], and +[@UBarney]. ### Nested Field Pushdown -DataFusion 53 can now push cheap expressions such as `get_field` down the plan -and into data sources. This is especially important for nested data such as -structs in Parquet files. Instead of reading an entire struct column and then -extracting the field of interest, DataFusion 53 pushes the field extraction into -the scan.. +DataFusion 53 pushes expressions such as `get_field` down the plan and into data +sources. This is especially important for nested data such as structs in +Parquet files. Instead of reading an entire struct column and then extracting +the field of interest, DataFusion 53 pushes the field extraction into the scan.. For example the following query reads a struct column `s` and extracts the `label` field, for rows where the `value` field is greater than 150: @@ -161,7 +142,7 @@ work. Related PRs: [#20065], [#20117], [#20239] * **JSON Array File Support**: DataFusion 53 can now read JSON arrays such as `[{...}, {...}]` directly as multiple rows, including streaming inputs from - object stores. Special thanks to [@zhuqi-lucas] for implementing this + object stores. Thanks to [@zhuqi-lucas] for implementing this feature. Related PRs: [#19924] * **Support for `:` operator**: DataFusion can plan queries such as @@ -169,59 +150,39 @@ work. Related PRs: [#20065], [#20117], [#20239] support via [datafusion-variant]. Thanks to [@Samyak2], -* SQL**: DataFusion supports additional set-comparison subqueries, null-aware +* **New SQL**: DataFusion supports additional set-comparison subqueries, null-aware anti join, deletion predicates. Thanks to [@waynexia], [@viirya], [@askalt], and [@cht42] for key contributions in this area. Related PRs: [#19109], [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] -### More Spark Functions - -DataFusion 53 includes almost 20 new or improved Spark-compatible functions and -behaviors in the [`datafusion-spark` crate], making it easier to use for -Spark-oriented workloads and downstream engines. This release contains functions -such as [collect_list], [date_diff], [from_utc_timestamp], [json_tuple], -[arrays_zip], [bin], and [array_contains]. - -Special thanks to the contributors who drove this work, especially [@cht42], -[@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], -[@aryan-212], [@hsiang-c], and [@davidlghellin]. - -### More Visibility Into Parquet Pruning - -DataFusion 53 can now report the number of matched Parquet pages in -`DataSourceExec`, making it easier to understand whether pruning is working as -expected. +* **Spark-Compatible Functions**: This release includes almost 20 new or improved + Spark-compatible functions and behaviors in the [datafusion-spark crate]. + This release contains functions such as [collect_list], [date_diff], + [from_utc_timestamp], [json_tuple], [arrays_zip], [bin], and [array_contains]. + Thanks to the contributors who drove this work, especially [@cht42], + [@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], + [@aryan-212], [@hsiang-c], and [@davidlghellin]. -This is useful both for users tuning workloads and for contributors developing -new pruning optimizations. +## Stability and Release Engineering 🦺 -Special thanks to [@nuno-faria] for this visibility improvement. +The community spent significant time this release cycle stabilizing the release +branch and improving the release process. While such improvements are not as +headline-friendly as new features, they are highly important for real +deployments. We are discussing ways to improve the process on [#21034] and would +welcome suggestions and contributions to help with release engineering work in +the future. -Related PRs: [#19977] -### Continued Work on Execution Correctness and Stability - -The `branch-53` release process pulled in a substantial number of fixes for -correctness and stability, including fixes for spilling, hash join edge cases, -FFI behavior, filter serialization, parquet leaf index mapping, and a regression -in `count(*)` with right anti / mark joins. - -These are not as headline-friendly as new features, but they are highly -important for real deployments. - -Thanks to [@jonathanc-n], [@alamb], [@xanderbailey], [@haohuaijin], -[@friendlymatthew], [@fwojciec], [@Kontinuation], [@nathanb9], and many others -who helped stabilize the release branch. - -Related PRs: [#20726], [#20791], [#20792], [#20882], [#20883], [#20884], [#20890], [#20891], [#20892], [#20895], [#20898], [#20903], [#20918], [#20932] +Thanks to [@comphead] for running this release, and to [@jonathanc-n], [@alamb], +[@xanderbailey], [@haohuaijin], [@friendlymatthew], [@fwojciec], +[@Kontinuation], [@nathanb9], and many others who helped stabilize the release +branch. ## Upgrade Notes -DataFusion 53 also includes several breaking changes, including updates to the -SQL parser, optimizer behavior, and some physical-plan APIs. Please see the -[upgrade guide] and [changelog] for the full details before upgrading. - -[upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading.html +DataFusion 53 includes some breaking changes, including updates to the SQL +parser, optimizer behavior, and some physical-plan APIs. Please see the [upgrade +guide] and [changelog] for the full details before upgrading. ## Thank You @@ -401,4 +362,7 @@ grateful for the time and effort that everyone put in to make it happen. [dynamic filters]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters [235 built-in functions]: https://datafusion.apache.org/user-guide/sql/scalar_functions.html [datafusion-variant]: https://github.com/datafusion-contrib/datafusion-variant -[Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ \ No newline at end of file +[Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ +[#21034]: https://github.com/apache/datafusion/issues/21034 +[upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading.html +[datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html \ No newline at end of file From 482ae210aa0a9644025fe83e4ffa4ea41d924e04 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 10:44:14 -0400 Subject: [PATCH 11/22] grammar --- content/blog/2026-03-25-datafusion-53.0.0.md | 54 ++++++++++---------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 1270076a..fc004d2f 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -39,13 +39,13 @@ making this release possible. ## Performance Improvements 🚀 -DataFusion 53 continues the project-wide focus on performance in general and -this release features reducing planning overhead, skipping more unnecessary I/O, -and pushing more work into earlier and cheaper stages of execution. +DataFusion 53 continues the project-wide focus on performance. This release +reduces planning overhead, skips more unnecessary I/O, and pushes more work +into earlier and cheaper stages of execution. **TODO: Add performance chart and short discussion of the benchmark setup and results.** -**TODO: Also mention we updated the clickbench benchmark to use DataFusion 52 ** +**TODO: Also mention that we updated the ClickBench benchmark to use DataFusion 52.** ### `LIMIT`-Aware Parquet Row Group Pruning @@ -58,7 +58,7 @@ matching row groups are skipped entirely.
Pruning pipeline with limit pruning highlighted -
Figure 1: New limit pruning inserted between row group and page index pruning.
+
Figure 1: Limit pruning is inserted between row group and page index pruning.
Thanks to [@xudong963] for implementing this feature. Related PRs: [#18868] @@ -84,28 +84,28 @@ to both sides before the join. This is especially important with [dynamic filter
Before and after diagram of filter pushdown around a hash join -
Figure 2: DataFusion 53 infers additional filters are from join conditions and pushes them down in the plan
+
Figure 2: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
Thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for -driving this work. Related PRs: [#19918], [#20145], [#20192] +driving this work. Related PRs: [#19918], [#20145], [#20192] ### Faster Query Planning -DataFusion 53 improves query planning performance, making immutable pieces of +DataFusion 53 improves query planning performance, making immutable pieces of execution plans cheaper to clone. This helps applications that need extreme low latency, plan many or complex queries, and those using prepared statements or -parameterized queries. +parameterized queries. -Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] +Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] -** TODO get sql_planner benchmarks for 52.4.0 and 53.0.0 ** +**TODO: Get `sql_planner` benchmarks for 52.4.0 and 53.0.0.** ### Faster Functions DataFusion, like all real-world SQL engines, includes [235 built-in functions], and improving the performance of these functions benefits a wide range of -workloads. This release includes performance improvements for 42 of those +workloads. This release includes performance improvements for 42 of those functions, such as [strpos], [replace], [concat], [translate], [array_has], [array_agg], [left], [right], and [case_when]. @@ -117,11 +117,11 @@ Thanks to the contributors who drove this work, especially [@neilconway], ### Nested Field Pushdown DataFusion 53 pushes expressions such as `get_field` down the plan and into data -sources. This is especially important for nested data such as structs in +sources. This is especially important for nested data such as structs in Parquet files. Instead of reading an entire struct column and then extracting -the field of interest, DataFusion 53 pushes the field extraction into the scan.. +the field of interest, DataFusion 53 pushes the field extraction into the scan. -For example the following query reads a struct column `s` and extracts the +For example, the following query reads a struct column `s` and extracts the `label` field, for rows where the `value` field is greater than 150: ```sql @@ -140,24 +140,24 @@ work. Related PRs: [#20065], [#20117], [#20239] ## New Features ✨ -* **JSON Array File Support**: DataFusion 53 can now read JSON arrays such as - `[{...}, {...}]` directly as multiple rows, including streaming inputs from - object stores. Thanks to [@zhuqi-lucas] for implementing this - feature. Related PRs: [#19924] +* **JSON Array File Support**: DataFusion 53 can now read JSON arrays such as + `[{...}, {...}]` directly as multiple rows, including streaming inputs from + object stores. Thanks to [@zhuqi-lucas] for implementing this feature. + Related PRs: [#19924] * **Support for `:` operator**: DataFusion can plan queries such as `SELECT payload:'user_id' FROM events;`, enabling better [Parquet Variant] - support via [datafusion-variant]. Thanks to [@Samyak2], - + support via [datafusion-variant]. Thanks to [@Samyak2]. Related PRs: + [#20717] * **New SQL**: DataFusion supports additional set-comparison subqueries, null-aware - anti join, deletion predicates. Thanks to [@waynexia], [@viirya], [@askalt], - and [@cht42] for key contributions in this area. Related PRs: [#19109], - [#19635], [#20137], [#19699], [#19702], [#19811], [#19829], [#19845] + anti join, and deletion predicates. Thanks to [@waynexia], [@viirya], and + [@askalt] for key contributions in this area. Related PRs: [#19109], + [#19635], [#20137] * **Spark-Compatible Functions**: This release includes almost 20 new or improved Spark-compatible functions and behaviors in the [datafusion-spark crate]. - This release contains functions such as [collect_list], [date_diff], + This release contains functions such as [collect_list], [date_diff], [from_utc_timestamp], [json_tuple], [arrays_zip], [bin], and [array_contains]. Thanks to the contributors who drove this work, especially [@cht42], [@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], @@ -172,7 +172,6 @@ deployments. We are discussing ways to improve the process on [#21034] and would welcome suggestions and contributions to help with release engineering work in the future. - Thanks to [@comphead] for running this release, and to [@jonathanc-n], [@alamb], [@xanderbailey], [@haohuaijin], [@friendlymatthew], [@fwojciec], [@Kontinuation], [@nathanb9], and many others who helped stabilize the release @@ -212,6 +211,7 @@ grateful for the time and effort that everyone put in to make it happen. [@fwojciec]: https://github.com/fwojciec [@Kontinuation]: https://github.com/Kontinuation [@nathanb9]: https://github.com/nathanb9 +[@comphead]: https://github.com/comphead [@cht42]: https://github.com/cht42 [@CuteChuanChuan]: https://github.com/CuteChuanChuan [@SubhamSinghal]: https://github.com/SubhamSinghal @@ -365,4 +365,4 @@ grateful for the time and effort that everyone put in to make it happen. [Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ [#21034]: https://github.com/apache/datafusion/issues/21034 [upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading.html -[datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html \ No newline at end of file +[datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html From b0225764033a7b7378bda7b4bfd9b7b0dbe95957 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 12:06:42 -0400 Subject: [PATCH 12/22] clean --- content/blog/2026-03-25-datafusion-53.0.0.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index fc004d2f..1576c15c 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -45,9 +45,6 @@ into earlier and cheaper stages of execution. **TODO: Add performance chart and short discussion of the benchmark setup and results.** -**TODO: Also mention that we updated the ClickBench benchmark to use DataFusion 52.** - - ### `LIMIT`-Aware Parquet Row Group Pruning DataFusion 53 includes a new optimization that makes Parquet pruning aware of From a980b7bcc19216d8ad1b7a933ecd796914bf204e Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 26 Mar 2026 16:45:11 -0400 Subject: [PATCH 13/22] update planner benchmarks --- content/blog/2026-03-25-datafusion-53.0.0.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 1576c15c..e78e6c85 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -92,12 +92,11 @@ driving this work. Related PRs: [#19918], [#20145], [#20192] DataFusion 53 improves query planning performance, making immutable pieces of execution plans cheaper to clone. This helps applications that need extreme low latency, plan many or complex queries, and those using prepared statements or -parameterized queries. +parameterized queries. In some benchmarks +overall execution time goes from ~4-5ms to spends ~100us. Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] -**TODO: Get `sql_planner` benchmarks for 52.4.0 and 53.0.0.** - ### Faster Functions DataFusion, like all real-world SQL engines, includes [235 built-in functions], From 859f29c481c539066c4ecd8017f2b809528ffd3f Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Mon, 30 Mar 2026 13:40:17 -0400 Subject: [PATCH 14/22] Add performance results --- content/blog/2026-03-25-datafusion-53.0.0.md | 18 ++++++++++++++---- .../field-access-pushdown.svg | 0 .../join-filter-pushdown.svg | 0 .../performance_over_time_clickbench.png | Bin 0 -> 57883 bytes 4 files changed, 14 insertions(+), 4 deletions(-) rename content/images/{datafusion-53 => datafusion-53.0.0}/field-access-pushdown.svg (100%) rename content/images/{datafusion-53 => datafusion-53.0.0}/join-filter-pushdown.svg (100%) create mode 100644 content/images/datafusion-53.0.0/performance_over_time_clickbench.png diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index e78e6c85..9fe4aa3a 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -39,12 +39,22 @@ making this release possible. ## Performance Improvements 🚀 + + +**Figure 1**: Average and median normalized query execution times for ClickBench queries for DataFusion 51.0.0 compared to previous releases. +Query times are normalized using the ClickBench definition. See the +[DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/) +for more details. + DataFusion 53 continues the project-wide focus on performance. This release reduces planning overhead, skips more unnecessary I/O, and pushes more work into earlier and cheaper stages of execution. -**TODO: Add performance chart and short discussion of the benchmark setup and results.** - ### `LIMIT`-Aware Parquet Row Group Pruning DataFusion 53 includes a new optimization that makes Parquet pruning aware of @@ -80,7 +90,7 @@ Now DataFusion can often transform the physical plan so the filter is applied to both sides before the join. This is especially important with [dynamic filters].
-Before and after diagram of filter pushdown around a hash join +Before and after diagram of filter pushdown around a hash join
Figure 2: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
@@ -127,7 +137,7 @@ WHERE s['value'] > 150; ```
-Before and after diagram of field access pushdown into a data source +Before and after diagram of field access pushdown into a data source
Figure 3: DataFusion 53 pushes field-access expressions closer to the scan.
diff --git a/content/images/datafusion-53/field-access-pushdown.svg b/content/images/datafusion-53.0.0/field-access-pushdown.svg similarity index 100% rename from content/images/datafusion-53/field-access-pushdown.svg rename to content/images/datafusion-53.0.0/field-access-pushdown.svg diff --git a/content/images/datafusion-53/join-filter-pushdown.svg b/content/images/datafusion-53.0.0/join-filter-pushdown.svg similarity index 100% rename from content/images/datafusion-53/join-filter-pushdown.svg rename to content/images/datafusion-53.0.0/join-filter-pushdown.svg diff --git a/content/images/datafusion-53.0.0/performance_over_time_clickbench.png b/content/images/datafusion-53.0.0/performance_over_time_clickbench.png new file mode 100644 index 0000000000000000000000000000000000000000..74f23c18d65c4058896f851cdf536d33575786cc GIT binary patch literal 57883 zcmeFZby!s0`UZ@M0R}oCAV_zPAR&#?F@SUq-6h>1qTbqwb$CudY*edaj$_X%1hqEA;!VLz_=$ZC8mUdaf1;9 z1M}YPo4}c&n3Dz!jB6qmqN0k@qM{Uv4tAy%)+QJjL^0ms3e4|b;kWx5Nl@e8DUCoV z8cQwAdGx*vB=*0{yrW|z@eMw8+8Al~9ZzF|iodSp6@kzjs_w~dwQnERoVV|qK3F+) z@)1PWR3ZK7SPC4kQRuziYMr)bsA+tNBeakeM&LnI68-t}?$KMZqs_TIudx38saJbr zJEy*R&dgg3Yj)|BnKp!8*IRZ4)kz2jO?nhtZhd&hCB%`OyX($X7Wold?~ZG3eg&-; z{S$evqlBA%D=s4)cTG@dT{^?tXld-qlb=}p zaK~Y2zkNrNh!B!=ele{*!#w+?H{pxLOd0o&7QgL_S{}E%qzbQd%1yNOruJj5_Z^C5 zwA;{{3+>m(aoxTzy41V}l}%OWy>>~h6Ek;>SzoKNo;Pa|5UHAphO{XJg24nF-^REW zV1a=J99;uG#J~pw<9g%=j9b8e5bzO8!~DOiHyG2d|KBm@y{j8vD2qx<1OJtc9864X z9nI~W?ouxf0*@NAP|-g7e0WS!?x&mfrdkX&7vw@}p zS7-SYE!<43HN`A!0Pz5Q2yt<63H-YMk1PM~@oz0P{@s%O83+5{oBr+6e>YWiG;t8M zvjO^a68iVl{MGpH7yoJ~0KR(lzlq|voPV7K1TBOk0RGpc3E{|_vx#G1h+s&Iy-;zx zwwZD(S!Lw7X-?C_uH@63H-W@f_>Zo&F$AWwGo=fE9&Bx5Ezu7m3ejzT$uch6szFt0 z{!v2wUf|nyeF#4CGOseAHluIw%)ZOjP-py%7eiL|H|5J3T=epx|^2BKhIvFnq zTEbyTghhdI?Y0O8=7TpF|9R*~^)Q=BsRsYfKVJsUet9U8u_*E1?f!LppbSPtY4#E0 zlRuyR`yQY*+5g}AzXbpPDmC*&*0&>h>Qk<=jTP&Un5`k3ME?IZb^n?I;#+=*Z;6_D zv8)BUr}sx$T$1xlXqDKYad2*_c?kCi^V69R9vnD9AVOd&QT8sC( z+c&?xF`6!ZXJb6aQC>AnrAXuPy{NBacN$A8ZKkw3LP@{UP_zA~7X51|6pDUW1YoGO zY#dt!c|n!WXu+K&Oh#N8tnPfxs0aT;MIK3t^aT&y%_KFodK?t3<*%e%znj)ez_s>2 zlvXhTE90>YfAoj=5EB+9%cv&>FJLeJLAQU8N{WVJOOe=+JGMkBN4H+Pox7{~e>@iu z3=JsUh2b7MNsmT`4xZzPG2xE&gQ%PTW85(cB9osuTdnggxM5P)|1E=m3s?sBC1115 zdcGA_D8sisk{5TdJu}s?@qF%ZEel@DQs=x8>pWW2aF{4t_lBj7%%oO@pi{}?H= z+#&q(tfTS#Px(8k6kej#E;W`GuwdKzXr^RvtI^l1Y<+>0yRpAY zW9R(v93n5S;C(TgJVZ|?ks(LZ!utE>Sab0*4i@6$thntRZmrRT4zdpL_uuwK9w zKJE3FljSNrwXHlrZCM8Ie2fk$8MAoyN5^RxSfI8#{fD6TdRP6Y0%Hrtm?nm)vtK#L zUz!+_&~A5?K4cQab$@jtKm)o7Ilj54rUPdq0?(9;d-@G!p-q@lCTaF;cA-gZ;g^9g zzo20|o^!r@xQDupOk??wJK7D_nLULj9ix?oyC}Yn=I6&uJn^_xCgC?b{>%KX6u=nr zB}hlp>;Z?t>p=8~x?T#$f%{6ziU2Fu7XMhz9F%iE)#1fXP4puvd&-3IXrfrR!u-hh znGETp_$U?3IKr0bfD^?ceU03D66NY0y9!>0dAh*SnrYnBEG+4rtz6fG_rhSYCMmB? z9*1Ix@4k)p#ggk`lrp*)-QHjuG2UboHZ_6>@t&Bp9d16kJJpN`O(i0H=IyTUC)aR=d#ltjT4f&+Uv<`*6gy(s0lrNQjs
    c5 zV|Ipr>}C|%pGM->P4)dov3?%Syp(=~)Pf#LU~jnrBP#R2p2Hi?>u8F2i8EUHG?i%xmCd|#ZWDAllosl9FuSt*^W_r?C4#q1?l1p924c!;-?mBcumw>s8P=!t*uzC!q&x|DFgRBSFgCaB%ifqL9$s43-aozpx&gP zvNO=FoE~s_p3_t@ml4I^kG7qKaT(RRQa$YZ2Qcmod_y?>R@sdO;%YS=4_ari0KOGs%}S z4m1}VvpoC8uX@qEU$eM#I@EM36f3g!^~|%l_ayXO5aDGkG^fw^eytXI;lXWkWA84R zRa7#%qro7^l0CK2oRwpvG!>U^oS(|1c_5PgPt&%482);`0qxb} zT&PYd2G)~6?D*fp?c=l*Z^d&giQ`G#A$*?$S2~cS6f6Gl_Hl>=D+tD^V(-okTqKLkn9mS%uj z9mHt{cSHHNKbql9)6kQ}fAu4LOY`7{;6-#Mx?7fjZLli~qF&!Bs>wI_l)Zt-)2`!0 zfIX&Yy^wSLdYigd1L%|{zGz+V_Fivv)e6Dgo{C;ZWyE5lc=x^`U8)EA?I)@Yjpd>> z!QdQ)*4dRPwt{&EpK*?%rL`i9K(t+dLo>YA6>=<30A*GpQ+U9*cRMQo&vhmw;mv4I z)EiK8?D`EbjQ^M$f{sMGA*04)XEIj zH;I(6_aVuJjO*1CPMsFp#w|n?O1OlGVQd3QW?}v zT=M41);!_Rops3r*-{_K=0fY9zEI5*^c0$Bcd;v!R^{f`g@`@3H&O|b@y0j)a1zYr zuS2Q#9@-G^^YnM@`w2 zB}Fbn0nz=CUp=PGbkt0du&{Ju!f|M%5MEr zq5g^|h$;OZKQq+`gLr}WK1H}B=ErFFchYKqOweC~5pl!Jmwuf3nDH;u zsWkmYTz9%%XM=?BR8We3cus|rS6Em{z^bLOxBQJHc<5+%!Db{^)uorcgHt0Rfq>I$ zl*RLSDbD7~iLJEJ=HISf$amftO`xNP3Sy-XW`BbV&&9IoS_*pXn-828rsgg=xlfdu zs3|KOZB176Fv4Hb`s}t5Zha0DDutgd(pUDd)zPq;9;}ZX9X`%6S`&kHBlH9wT zq)yaC#Q1mkCK8Kfd07!{LPzsgI6wm;;xrF0(5)N2A$+_THJT|K=Y}tcagR)3+-1^& zzV62dlKCG?PH6$Hx5&K)e9jigJH)%QQa!pWR?__Mm(HDUyWSg37QQ_0r$Z!sjBfUM zb>kp>s=`X0AJs{{WhQ*83Ejwl|5Y;di6(Qu=zU(Sh*E-xP5JV?l^AuX#pTIZ(>(my z5)HdtuQImTL`5i3pi`K0mCiP+|Hl@dlXZsKjS@pgNb&gjtEsD$9p+ z^7YqyHmd`lT)z9=N_4^R8?~)nqx0M!lw9aiRmUDUVe4_(RH&Z=gok(9qjzEGxm6Lj zRXJt5Tk7;0P=u*yM&(UZ511!p<$ZB1t9C8=b?n@a;SH|G4>+{6)pPOFPp

    H48JDE>x~$n{5Q0s1x^V`XMX;^`#WfxCu2CrCJ--2k}hY^c2WpkfSa z^@B?;u!xFsiLfpswp2bFeYi?@k(%Yo=3ZW7D}0_dZM@_1rA!^HfV;O!eHh zAmw){cb)SdoAug*@Skp$ZDmBsJCG^f(=2i_>3SxQPOO?JGppMR;hyYte{=F0d%1M0 zYHk)V9F>mKfez;7x^1%w)~abkQa-0_)tItAzMca=O!Ta$-NsB^mDS4WPLmxnOMxeI z#wbiM`L;T2{D{|%K91lSg+I=*09&df7ptrFixoFWiY1W7!su1cX{m6 z{q@yZ)9E(<>3Uv~_4ag)e8xx^ALymOaI(oH6nh24KyBhY?sH;yZGM3Ge@ zyz8T`C$%}c!v88mB>{BC@4B6AQBfA?I{Zp#J^rOSdFIH3?2epZT0 zqUX`YNRf}KS)fg&NtdkkLI_W#-}PIyrJx(6H%yaU>rs!N;HL+_TugYOtP@>g)WPRh zYB$%&?XkbArCbm9y>MSK$jrRG$TgayoF!KVAqMMh39KYLS}yH0%-k0~dyQlsq)9Zh zN*Luux;1;4uVG^Ipgs;ake0_mJ-V=?3$!X64CAp%(<}n3>*tg-MMKC@@nqcA8ui4W zhNXv0Mbk><)+f?i;jV_oDWD4)kX1eZQnd24lj(sfhpg&yz5AYy6ZQbKJW3f-lc6^5sK9?7M(ltVUK z-k9=yO?N|924m2_yngX5T^Nh|PlnX>qP1nMez}f`$@~19`cpPlDR_E|sx^Qj@`Kf} zbuIk`zH?^2QvTT1tq_Cn3h|Eq^6B6KSApKar1O~WjFVUi(gSw6b9pYTUU80|n=6<> z;K5W-vnY4wZ#65a+E$i3)L{aKpc%HBMNrhr%pGvOJ_y_Ad_%A9B-rVx{Y?ygV=Jp-m4%3RRfBeE+!$YY9w5=>d=qcHWjK*vWhw$ zn^>xO%=&M*16-=1C zC}MS(<8@HAjclz@K+m}9p^uMOgfG{`-an+47=MS2C#MV>Q2?pGrE6YZPO@`B=dIHt zRUBr5kO+t)Ty*9#k&1cy!f`c$W`o52+p=Y1H(SJ%=fX{4^gOEz+~*``Lw!V6IeX~` z;zEUQ=5y^1jmizysLC4)^kj2x3n6S$MoQ_h1UY3RydCWKK@CFM7i5YW1E3CEUr6ek zb!zu6ORq=nKt@{t2+fF~{fed*jGNiVW!}vioG^<_p8q`1J+|9kY6Y6=L3j>}B zF3O7FeCX*>PSTT_j#zu4Iv*@x%{}Mkh%To8w5Q;E5`{P2Cw%Ee2%hjwOwFzBdL~V0 zKx?nd*Y+yK+xw9M?kCWyF=&h?S7x*BXUm-#(Go1SiQydO1uNZkKW*I0OF$wiY1sio zI=!Cw(i+3ptKRWdh#%*IIb$?h;!ea^Xw@vbKh3W;YrK5R1A7WmHty~xmwgx8#Vm>2 zV3<}v5{G&j^(H^SL5~nd)U7uwfO?BU9C^5natTiN&^!$cXrb8-{PfQD{YK{gJR{{u zQnl=AhZplTO~(}Z5QUo)aLz2$&fynk;wx!#*cp=Z*qEIiXJlbkG(0i&-A>Qk> zv+)R@@xVR8f<>{8D^%ve7^m|&MP$L6B;C^IZ~EoCAypJBD(HrGn-O)y>0p=@Uu<~3 zCo=3ja6yqcJoOaRcJCOPw4EYD1EpLOXNxl+2$j*ej@h~ zbG{@(M%o8m{v8TS%BkJjNdAMgW${FP&74r$f%PRwdL|YHHfgHA%l^Sj?P484IT`)% zTQOoXFz*XW6)HHoSj9t7Z2bkVHl8c?I}YqJ(9HgWe2ztL2KHA=mlx!Lxea2=6{m)J z4Ek%LXZ!kWQ9N-15sR965c2Zl4`S->b24(TprHFEOp@mh@J|;7bNQsgW5OSD&e)-# zN>i=6>A~BKOL?Bk-f4{y7B<4*(XJ#VGCz|iYH35UBLePX@4NtXb_ z%zKv0y6xHx9p6q-1$40-W=;KCE7c%PhV*mwIjd(ZJ8e&bvpnS%3Q!S8Sr@UzL{Fwl zYZR95<&-+ZW%rR9d8e{V<^qJ&GpC?yBRm(97}%bmXn%yK5**^I*X^lenA=+2x=R_) z7G+eR%II<;u(fj%Tp*!J|rwRFG6KCH1Qt_Nq6OBuHHzE!{5P9ns+^E$F)#@BI>V% z(U)YYv;cWQPWK-)5kZLluKn*TS<&kTW1gRmHiS}IIbnDnX;EfLh*=Nh^=wqc1I8$+ z&-MFhw7YEqRx(8`^&(l-fmuf$KcgPd;VQz+Q1_!1<5nN%a3&;Tv0!_Eyy_(b5pGNh zurg5UzDUs+qJ+`I<6~S^`ny{&&p`T6Xqrlm#}|1zwT!%|NlHqkmiX3d*c3QXofLy| z%Kj#x*D+1&)IU0c1W_J1iLC|W8qCQU>L!~uEXWw~g6d=IqXfe1GhOJ* zwNz9~V6_{r`pyz#Kdrl#bn-rCW~5AXL{@%%0?k18;#PL!{&~S#cd1w<+tZ(Luxzq& zLVcT)f|Qri1xM4i(LmI2Su7Ew^0agmUCoIlk1ebY2Wj@Q?rzUw8)3PVH(qdD(Sl9Y z%qX_lx^jXL4+pFj)a?xV5E1vxsQ(Q%8jIWM5CiO-NE(1ZSwPh z1<1nZ6J_CKEXXd`frfX6&;dM0{|TwAT-?w6)30F_mQ(g6kPw9dW2TTpzrr5JTfSXeX1V!`qhJS@bW#uO_{KbOV*Mz^E#au$o_w|--U zMbQh{igxJJP~pR^5taV5uDw_3nPSsodo7ZNZl#Vpk8-$gehaV)5wZjM&=%4$D`V<< zFK2Dbh{YZY4V7=JEH9X!zij$o@U)Q7>8(Mt(dF?^Wr+XSS;=H!oR9+aUF4ViQ!V5> z=S#-vJF$~e;>$M_u{EI4`|c;`)<#?;k5HIb9CFZ&Msbbk+l6V3T9AAFV++74+wL#EJ&lB~^}I(!SuX?Aw<=9}R^u^oxu?l~BH zyK|w%)Ch83de7>|KU3S=EWrEWcfe~biV?vKjj`*M+#TWZbk|;D=8Z1*w3j0?1^Q@K%}-2=<*;9J z6nx88=Rn^7Hj78M${59BOJ@KQ7$$3Q-I^S6pTZJ<0G{*!kG2X_42oAGa=vrS6*YT# z#bdqPc}Q5F+GQVZ^VCOGbt|U?lzeu-24xuSnv0MQa?EwPwTHAf)I^y^TJU)=?VYnK zJO@9&x!_^XN<<)Vs9eKE$`j*m&%9cI@EVcuz9?`0P(N0l@rK)BH#JAAN0IJfpQk!q zY=rz9NbuBy6`1$;p1ZUCI6ab6^$!x`t5E3K7{f$x&!P`b>{8Z(O!lz%=KC!DWaEyL zPZDJDzNb@ja>B9MDK)cMO!OsoEtT{dW5&JyV_%?|>b}t*Ijn zJ>7jq)DbX(KG)PsL3Igh2#5Yv9)MvzV8nyHzW*zRyFr;2K;iyc7o<%syTL-2_)f&V zeATZUmcPS(tCZMflV%5$I4y~?}3NHz3)~gt3+ci|w(S%7g zEn6B+2ax-W-#D(msI6b2QIdIwb{;9$vTGD}D$1%O^geU&=$D)-&wnQef8qZ924o=) z3yiHlDZu!AMwqC-$dBl*tn=nZ^0p5SV#LXc^%cAQycm6X4>QN}*qaMG^(OmVGzj|4h=tWm#X&Mu5q?!P2Z+Dgl`MG z-?n;&$0|&*(+#AeM)X|X#5k@{kI$LvoU3^Lr|6X7^O#HDXbnD`O$U zlVx23OiOFo_k8PDpoV0y6;yrf)7*>Og-LmC>jACyDsSt!b%)MB?aBy;u0D%_StP6K zU_(zHQjeW-Y9?qYsJQGjpliu#-`by@WH%Rkk;{sVBCNHtROUp9qfCygc~H9Q!VNP) z!D}e=Lg?t@LugE3(XIrvqBiWz0c#Bfrhd>I<<-9q)8is0@i-74Sqp z)1=)xM#}TEu5GtXN-ZUrrP^D4Uh|VmPbQ<|I#+B7tUn=c*{3)M@)*^(G$3IklO`yk zJ2?IcqoO282Jl3>c+m9rx8`yX;34K5P*j}e0;X;Ng9kFROXH3LL1GaeZuyH~GCnR{ zDAWx^g)bS~zveb6tzEZj`IW+^BWcJB+#ZMwu0Wy90p$Iqru z3VoijxPQul*T)t5)OQX8A`)0x%Uc`o)+76JOk{DdSy0fS3YblLwBmcSNW$FfK z`7PZ;?erQStU+9c+LZ&gy^R|y3qto|bBt^jQ9uG>BF0t6I*QEZ1GRyQQOG+zR_sUx zBreo2tJ3#*XaVs6pUv2;o5PR3`Qgi+K`i>?oK(8Q% zekbj_f7Cl%82x0Y7!*27Gie@02yVH7JbIo`PanuG6w*HiKMhbF+4&Se=FmK8GAg`C z7Yk?6BVQ&DjOhY0{OdsuCV;}oED^BjMU-zCe z;iKh^wVqe;Jmj$~zHPf>-e(#meO}c98cWid&PwO_zV1&fSS4%{@q2kx1-+iYvv9&Y zb1N`9LP*SjEYDo+OQTDxETblg46$46;U7W4)RZkdL}Uu_iy|*$gs5xDX=G&XqUrki z-FLQ2d)S1sN9$U-y!(s;o;XMqSn)+XC|n|v2q88B>In{Da%uSWCT0gFIw4!{ozxWH zeoh}l0r+eaJrsSQ;)bx1?fnYRyWIDOVQFLQ!BS0Xah-&7?6w{*V32%q}!fuBzBe3%1cziqQdEJRDT(u!R(N8~xSP^$~R_Gx2X}G1U_i;0tQEd|GIapui z#**(Ka2;88m6huc3Z1{9G*c0yR^EIS$RIs~_1Lg#AEqX;YwloLp?Wjz5A{one%YQ< z(8R84#?WnN$UXH_JnD?7&-i!)n?WQ#z9gMu169nbX2UMp0_&-_?v?1wIiY+O)MYt@amWIX;44+PmMnsIy&zA+=6k^mv)CSL+eh zIB3iD{HoVcz_CgnF-|Wy$#J0K_J6h|` zBfrvD&VnKD0nbuV1t|V$s+{5f9@FnMn>-fhf$j#ArB(rA-m;eCrk@N|8sm=EtfPyoM=|6-v%U?9KhUdUS<@(-+xtD2@_ z1mK4;mug$!{|2Q$h$YK+x(ja_OC+`agazxey45btG@~7?}xw;K1MGg z)rax_D5L*=+@q@mK2=?e#Q)&CrFq?^P>NO1d!_t)75!rzDl|aZBydxW_|H=Gud=Nl zit>%r|JOn=*NP~x%m~R{n{2Swf^yl}P7KEc0`vBs{>zc7TYEHypA6j>BBibd*(nMP zKOO9+63a5-`+yT53UE=a8HVnu$n$>mUD~Ie5Sfu+2hIm$ED%-!IvalHT6?(8Y6Jo1 zgKQ_v;E0&nujROyF)!Y%&BhFdm1F)7shY?FQnjuN4gBS5{2?=uG!8(jY7)YxV}Xgp zW#GoA+GTWQ`*~^!-{5uxAp~?jrZu1|;=>m{OyWAuP!q#oy#z%df+fcFvuolyMfLPf z^$p$f(vm*D!MjYr<-Pj^l+IlmQ6Q&Zl9B(8k}pl@%=<5*%$L}s6EpUAF-|| z7}*>jv%I-wGuUyNSQ~55@veARIj;B7l^Wfk40j3EsqD2pob9(fv9{qlzp3$!KwEqN zZBXxMAp{y%Yy=%Y>XM}ev^HBG(BPQKniBAv&hOs&>lJanZZkeTwWIB1+n<|wN2pek zO;jDtG#69(Sx@RBOdW+G^3-%v5=Ez5q8gc2M8@B zMx=I^xt+$NO;rt97d8<t)WlC~jg=3=Cimg-{MAj?b}w&9|H+QR6yqBH&%6 z2uk(!)7-TCoI&6scbi1m;uF>tIxtZ9IT;y4HXl1&p66Sx=cyg6SqBNchKuiz#!bGU z-Ha#X_PNF$4)_|1iGan+mg%9C^X`-OdK{kps2G%Bn_IO~PTBBTtkz59vLQaDA~qal zUWOdE+>!1H5jw3GKO{1*l+`OU{l?V{j%XPo=2GXe zZl3zc$=(XwWhNype3AARJA+u1d;AMk7TnVI%zUnoDuBGwmT%$oQX0pforaO%s<{Jw zxMi^Q*eGwQM%cC~BvHdS3Nl)uvKvKYWDH`!0#Jb$Fp*wtJq!BKM((zB>b}xd6ZQCp z-6C?PJZD;?^b9ULGTb2R#)OG_4L6R{Gsm*o2H$GX>2*DV6u2R4R`ITHX9D%rtcQwp z@5-w4J8RKD<+M{1E+}4jHGXt3_J#lx3fO7?#L9m{~@vf^ED zzS%3N{&=MvET4W2&|pK<>$XVRPb>~nQAB_7s(4(!C&tbYt$RaDD7L`@Jvc`3>gC>3 zhj$U>th6WyX`;QJ@7yUH$%`7J|Bu0mAaQK9Kig_87K>U|UG6{o)-G)5GNAaZW&^AZ zJv|A0WJ+SEb2YHW!dnpjh6)?PyT?DCTc43DIbZ?y_hl$l&*1nSX;@1;t~U8RfGjuY z!v~2_Yi4SDL3&!oP2Q{y%a3z5@PBEhbC=FlGDYzL=Hrlvsk3n^*BRT0_-qYi6?|g* zoP}dE0HU$*=nnt#`w(*Q$LqiwmOS_pRa&yYFF~TkS*e{=c8`ZMi7RguY&f`v{8roE z5|z*@+lD?|sAt6RvP6P=*2p|dl<=6%&qpo-I3!KB&e(uT$OMbjveTRlb`(|)HVIjHxChe3%GiY+H@XZW}if470H=`$_U_t39Q9H^i_&mt<;?zw$vT z!iJ|uKThYo`NEl{sgUHHLh(XTYvwcl?#6JLt;|bcP9!AK`l=P@jvmD-1nx4Ja5K)2 z<(PHftp<#flg5t^4rven(5j#b3<8FCn@d8%2=ygh+;pz(iF&O|h# zFy?65mie5GlHP~FS#Vo`(X>maH3EPT^LIcf#v&5M>4}>LFcd`>ZW|md*30J#MR%9S z{1y??;{h(vsppjoc&ICN{a-%Ye|iIov%rGTQ+Jsy_>T@A23B_}2C|{2@s1Ub$cCH# z)l7FsU@WNf9uhl$5LtVmf0r?(F^vuo#M(psV3r1!4F-Gw^UuHi)F(#O^_AxB=qG^B6S)Cm z0#GOV!L{~I(E8n6%xkwZXcd*HOq`u7?vV>kUlAClCCzG$g9yRgm%$o~>xu%uM#%ZVuT*U-mPOML z#H60DVX(K-&vVXRaI4~q$@SsxV?$JaGLKzq$m3`2vmP6)b(^I<(<-mA<712?#^#C< zm#*dFu2UnUXG?xEmRf^Tr2v5v&@G*s(pBeAm@(oazFr$`4JM=cE6atB>irklrF=Dh zTGY)X{vwpOSNV&FJ96lFLMw7fY)ca#W=%~#lV`@diDG$hZASYfNg&w&rC;vLN?Iuq zbU?k$F+U@^Z_qJ4mGB@rd!CQ@ot;SiFVtD1GUq`DZ8!j z{Sw(6z9EDwmmoQ4nNqqEuzf;fCv$;K+crJ>2)XoY zRu_e6wc}c_13N2un!r(S;pg#9&v>uC(z-mNtkMOw!W;)u-*qe3*Y0>=P&CTy8R#^$ zFpw*7XZ^&3TX@W|;qomy4mA|Uvd(`O6NYuxpw_*hV3;C%P%ph%hIpShy;9JU>ObL_ zuCl|oe#37QJGw$@EotVH=q(=I#U60e{)>uPo^k!_u7zM;SahP6(kp_g5*^=J3(~$o zEA0sWz_nTN?Ke1Vv%>Ma2_djMbLeL*rFB0Nwo^!XltvW3D4^Mv`b{~x{UwA+Ea)J6 zGN-477i>%$L?>j{Be{7sJKE6KM;)~2bGBix0YKSV9ks-q##?{gP$eGt^Z=Z z{m}wgm7e-VYDSg%O+oDc;p(qmL|_j@GFT!)(}@51j!aSg<=zBtlWu(VPXPRBH+s24 znX}H-Q{9&n@=?Pdkj!fNolG`4bWBrgvza*M37*uflr~0GXN`fHLK6 zPAFzXSwCmP(_gPAB~O7GcMGCv5;K2{_k1J2Y!l$pmS6F7T?Ssr*o{|nSD`49pw96%j+rDurRy$ub;Ql?O zj;oxHT7lM5ckTX=Yyq%G&}BU*t9Ea+K)cQrRuf|kM2^Er~Pu4hNPYPz~x-BCo`hOM`ebnS_UUV!yXUgd|y z%ApM&2?#P_fRtP~4eX5TWotTfNMbko(Bpda^9Q%n>OizN3JJd$p7lAksRMY!n^!bI zJcZO4bshWncxJTb-5Wp4k@979ki>C31%V$!wg9*Ja385KWtwDT3KYhwjmm&rckPDN zSYZWl!=P5Vxxp3rEveMIlQN+7hVc7LORHjzt978nV=+R~x-Xg61>hjw0@p3vB6Ir^ z^BH|rUW0Uy3AhzYznL-r9Ku2zQp?AHTN3j#zMFf(IO-_#ubrEa{Dv-n)ZFDomC@&? z_Su^L$|rqQIW9&An04^!YeFpEayTKu_oT1kpOKn&}1W)c`f7qv0j3m&^l|3np!7gu0%0 zMb71R-Hs4FD5&B@VcwFP&QHsGWedRS*O*J#^vU1Xvr#lML2Cd8uWw&FC=)Y>Xey{m z!u8l^A2J5LNa6U0OeKQ_CbB8n*4LZo^x%Tb%Q3~wU|d3I_mWIDaWSu`u{2ERq!XqG zIs{aRoSs`cBO1M!_QF4TB@OY0d3%2isjCCCilSF=F^qxd)&m8%iM@W|omP%V2u`g9 z4s#IKobRP~)OUK^_A7pQ59&KG6*I}-v)%wo%*Iv@_?efbw6d}Bqsr0J8B_ZH_cq%I z^~!-c?|V;ss$iHU_7ajc$`>;WQ0YytD5_NHlgv>82r+J0GzTmgrG<|-0W4DsvoUd4 z6Dh8<6J)+;_H~PqvdJ2`L8zxAwW+yqfoIAUBPP*{!n?CqoZc9~lm{Ea0EcMgh4b1} zmEH8ltNEOIv9Wy31uUp8|6Z@Nm8BJRsl%r8NH8vTwB_)g`sUag!mCdXo9~OT>okI-#sx` za4(uagN+y_GP<5RLLO}+v%<9fm88oFG7(2iI>^kb2~!fD%;O^8^E^E2hLvWX6- z#Z6h&e8fY54drC8Wyy+kva*}hob*&Zd3ssMCp|Fi5nv#QTAZl?R*J;Y(S3kaPw%?= zULz%YQ|#492kt3UPaC;J|3VE}uOM`6%%eKn5mS{r`hA#lLo6yjU&!iq6rlx^ta^uv z1u>OZ3fNjSTI}&ZFWR55%@eEY=gJL25jdxDtxpTARvWM?+yl8y2BWPYMzPZ2>l0;J zA;etW?n4$cS0UIKPzKx1I2iB%)bsTuL`S)zfd|oo9yh`TU~^>?aBN+XJr(RhJlG)| z?hh?$PdD_@;+u|j{W=NW3$@)3L%YLpXT6vY0O@A_{i zNCX=+pkfF48bc2qpi#1{EfJHn@%rpo(3b7|z<>4wD)8(p`Nema{lhh2tA)Pr>Zt8L zJ&3Do zz&Ab`j<|VMrJD{!iv3V3xh>~CV|ky%He%*fbFpN%EmvGTHc#A|%gsWhog!QTQ>$OR ziK)_VP6$2`YXarc9rHF&2RyCV1F%cx!A7OeHQT{-@shlk$B!gLpafdNxzQattMMC< zqRt7kTJ`tNFT{By$%FuEvp@4qKQT;{z>tS+3css-eXF|^=AeP3hEumno`=A4!H8W| zcn%B-!&n(+ z7=%@L)}(%;eYZTNRKXSJu-}_?FIYvdi4B2@8ruN&b4(|uwT%)hjj+eLFc*kzUgbSf z?xjY5iio&!c}09~%cbnqR=`-HI`Z7^qgL)kwgk-Qiw*aUZ~x6k{n|xD(&Jse_O(t{ z+E{tePEOf1*{r~Q8rb*+|TU7 z=3jf?_XbOx@^xy3<5Fm%TZKu*bN5%4kWYU%LGxrR&KWH$L(ANdu}kW zUJsPSdYSv?jp+;!zWL@2dfulmG*vO}>6xu*-ipX$og9o8%rX-2K2m(sdfj8i%x_&w z)$=ETE8qIDbC5K(%4qsm?7#b_GJ$VmH^zN0&t-GvRS>@lHzLC8%@;eBw%z7N4@qtj zykwAe0GoT?tT8kzdthhia^s;Z=uX$d@fT=hV^2>vjNMA<;uKErlqZb^TPl0FHkt!FW~+b8c&U29OS2_w=ag0--yvEZ##21r@6M7Z4{=}j(gV$`^rH%9d z=2SUjuRb0;PkH2N+DZ`~{N?S9JDUKL2c|dFcz1HKtYiY%AY{`W&A4m2k|avC;`ZY4 zli}gv_;n=F-H1~5eGZ$e9e0ZDSmpAx$flm^>p6V?o_Z1J?PS+kckb8UMUprzlnTz| z`POI%FRoT1H8(nXa^JI(D_j>Z0QQfbA5Qf{y>{E=B0mH!{6B=f1z4187d9#*D2ON} zY{eK+x&#RUfl*QE4gm>8xa3U4??3-J`?|J! zU*pUhPpr7tz3yk+z2CmGw$%-?sO$l|?;Z9x9$z=bd3&t=igSR^%3BV+I$bZAC;8?f znceZWk|#Mq@}k{3E|YwnTma9>$Y|PpH;-1f(U$T?l3SIN1l3N5=*CoZKDW^C=Brq1 z!l{a1a&*wEDSdWk++%%N_vJyQw_ukZ^gEAdbyT`6QghpqT0;ILGT)}`Fp>AJ2R;we zG9;4GHa_%3wHg{Z0WeVI{M{nWRRow|S6WK!*hlCa)->yky zq5jPC@QeEx9y?{Er{_cR*}_@z^D@(8mH}we)xJpb46+rRFW?IF0gpvy-co%`P=4J# z^gbvHG9S4cy>5&BZVvVX9?%`RVYW*89ip`Fe!6O(-B2+5lr1|KV)Is&0fPO<^^gue zK9iQWAt{Am;3`F+d9t|MFsd6K%<{~2ze2cFr*84q~A=XsylT(`irI^ zC$=+~2B?xz@toocsqKx5dwDU?)@47&TE)sl_wh-DMgDun8QfOVlm*>dGr36EjC8wo z8XY(e1poQz&!2}@)Ba3C(^z&4WC&SqYfInx^=kGAL&}~B$xC*%ayz#UhU5ek_k)#T(vyoI$U4`~1 zGo}lNj`F-E@xZGqmon%i^E~k|ine*U3fWc3zey8Iz#QDp=5{jSTiw+$_2gN`+T_xv zaPBwY0=bx;O5JbMd=A9GSZ(W9MA@RI85O+pg2U8W0E%V=y_K^uZch(coeirl%qkh^ zA<2Qp&~T|AGQ3Ukj?)7U5MGrvXwoU!_dy?8%;QMX?{8;YVx(o(pc^;lOX@vc*O>=Y z%<>n$64EBSyFt(Q3BUsS+cpyP3pOuK*u16qXvf!H>7U(rTl-2PJB<^5W03l)fYE1Grd zC2Ylxr(B2eRc_05l{06@B%f&&82XApGCI{0VU*o_SMGA$MJlMkRUQ^wX&!)tiD@6c z+-6~-dnOgSIS=7g9w7-4MB#TRm6RjKGDe3Yt1|6bD<%I6aZDBe1T`_HW-0$*a^?Uf z#!o61L0v2ep4V4C8|e5Q&>wT88(Q1;J%gM~`&Fk%{CQ>j5ZdI~=B1UTU(gBeAV+N< zk@#Ew^+oyAp`~xpFMbrhqO@(->3(T5D06#;``z)ukHZ>Eb#zu2 z0(i7fMDcToNaoI&4*Ab+0fcA~f@zT_3K6kX>VIXM`DdGL&KJZ*pX9Wu32N!7O`b9Q zDrsV-p3I?qIVxGzOkfPs#%j5nPPC@Sa+Bp;f*vyyD4z6!w&Lo;a?9B)b+h$B-{bFJ zl`WSm`c<;{v~6zokIc!~NdD~6o`E42=3eLtck>r^A59AT_MP%zCFbbigU5fA$td?! z7MXXq$U1eFR*2-3jvO2B$cnP4S{jk-l4(B)RIvtSMbvaxo_;^C>4Wb+1zcx(9kaBK z^2TJu8H|qIcslMyue8r4P=CWRGqa&Gl{M7*uNv&U8AgNR@x(pN-+x$T2|5$!jdQdiiMbsS26O z#bFSB`gRgG+tZfJy~Ja2;cnNhM}k-tCsToG83*z8iGrp+UMucSq0oWZSKZ%38p>`+7$2z*b9^k{cHWCMJsg6cbUHrd@kz*9cl%q1)XwSY1{ zE(Y{SLm_{X1LGoNfRdR=`oJuzjoPkV8M2jzv-j`?*6#xu77#ruL@UaBnSnDVN_mSy{Iv^36l7JB#i#PUfzAY2`Mc zR@*ZQtwrSxIMac?Xr9|K_~@Sj|Dg;d(Ly86t!%an5mTz6zPOFxBF2Ypx z`Mdk6?&U{v&*ob)qed%6Tu&6fnScF&wEtt5s7z~-!ECN8^f3<#tC@vOh$|nRtq~qe zP3W65l(p&bYmF3|Z0K~c9xFO5-^h0F|+W1AR{Zm-Rhr7HFv=8#yYb$?oqjQbc3q7E`%6mkcYtnLR^40!RcZHvPf8G6E z^|MQQS&~%BZ&BGYm(!|$Za=)9uAaGaE8jgb64B34_S?*1K#u5H1r0?9F6q^O-WX95 z<4G!xL;YYwTn)8Cora|o=X@9G#(vx!2%+R+>z->Ub{|F376yT!;v-m3hqrB(fk_wT zPAk!Tt*ciKn7tiv-RC4FHzHZ~D6pe-ysy;m8^gyd5KbQ?$$b4{JYo>g9$Qp`ZjPdC zTcj_@WfVZv(6*eJQO!3j$jtTbA&-S=f{`^z!V*7sfTJ-WT>YP$m+Hk%MgLNKe%C26 zUB8UC^k!@FbrJr_qe7&~E|0EVLv8LG1C(aUY{c|~_wUc^7A<^t$XC~&{wK&hMn$kn zo+#_!g-Yt-;u3!17ja?G_)7FcYGze2rwcTfaVi71TcAu(K@u$5bW@Nm?y_dOOT52< zVUPg50e{<;ni_+wr#-85nM>;N-7s$>?iPY7>^S=LW-);EfoHOFT=ld>XS8C|bADN( zcmG@1z|A47IGs3!4Ppy<{TTx2aySrj#Y&9%d1=s*e=klzJmd|;6jA2D{TqJ~%uCRb z#y&a~Z~H$H@xg$=sLdw;GF`P?(qj5W^oZj7x#?@4=6NaqiS3#90xQa(SyBaP77$C% zUPs&f5x8uy))h$g@6EC1B;v4mo*4$-`^Z#~B_AgmA~KR?dOBXS*j$6# z0<=>v=|N4j*u0y^3`&$HDybA5m%k04_w3|XCD?rJi)Wq?^mPPd^ zfthZ+PigZ%>t`qoqwyrzZlo~}sg)yD@UDP{m`951eivsY=y>itt^}bPlUD1^ zIYyJ8^}n#{()KpPJ8ZjeYA0vIUZiBE3sCYJ{B-?PjAAQ5g8E_F4iO@N`!9faLO=h& zJ$<_KhBDVoo*-PJjZBL2fmBD@+?{)mkR7m_^ZV}oz^{H~U+ zyz5fs6j6!})ezq%)!*)ChzO6dVDl+h3P$5#>34st^^uMG z!LE^iHuk}5*7uPh`(vR`){=Z#nk^G4Ft6`dvbVS2z=fcqt)Aq2z5A~6yJ@PBy*@#X zqMrbxyCon~_3257&`ItubLNpTCA?+Np@3xoB!DKZ*gJ#-*U`!AweD-|qc-UV6X&~l z)HK{^)PD*nX!O54R_^5y7p6L#X6{Y?xFdMscPpBFRzac{YP%=@gw@3!?NsTy&|PZx z{USL%DP-Zn#43dnOXZ)lepw$CcF5EQSGxd}3894}0^vngRz1}f@Jebn7HqRAR}ZiL zd5~=&ErPm5UVn;U^(>m@>`8 zDb_l#JgQrV$dfw$?$gXWh)v*Ml~BhIs_-so1al>c24y+j;K=AzL?v|uShXZdYP3h; z`418}S&zSKuwClK-AcWqF2%bb;8iCn5PDwDaDRXbEGKUYW9qd}kima-sgu@1wQCd{rK`brY~;qLS+@BQUP z4DD}WrFKLLSe=ZkT7_QyZosj=Wp&Oxra1HMJJb&Hv~6mS%-9{;+Eq*R=p3ntWRG!% zU$>_2um8Q*kZp(4LH3(mBrl7BKIh9XsMdyFknkC=;P~DYXJye_Y`{Tv_@U#bn}4I112^U>bBQ$2TA>YZ&*|RBZw(FK=XrQ8gStCX=x(_}^{QU6COJ ztvsKHnZ+CyISEx~sj;?Q{ZX;3x+wTwm0Kq{pONGAL!8V8Tz=rXUfr{!9r~@Y@uaCY z^6R4DVbl27{tplE8@q>o z_%Qo6LPz}!^KH)RKHe*yaWm52UKzN9BoGPQN>Mb}oFRGClgkX_DxpSVN@o9f74$lI zs9#ON$G0Apg#>(x6z|RL3vJHfF>b)ouJtYPZc2nay#n!y&zl&fopfR&T4_Osr_3$L z=|ggrC7*V=*4-|Oh?LWeV8^kPrx80`XZ)nKS|cT&dw{4=kQVicyEA8*j(K4NLqq6c z&e!gG7}dre-FWXL7{vTGm&bn)!|aKuAWVk`I=Dj{A2(DEXmmeA(_e%Mm2X3&~bMU0Dny! zzr|leK0c=(BQ>=dXKoKRjvL|q%T&s0ZQe8!aqt@Uv?2e-3$|!b+1qkPmg&a zi=ZRToOmpl;GbIpCvj8X?m+af_vYiEbA?X@wni?(PBLr^zXj&K1YxmZ_n$Ndoj8ez zhop!HEDHl*E7PDMG)YHwOx=*MBU{@HVbdY>FO3grmfR!~(QVnE^RbotyIPs#b0N9X zggh$~Xf3sw#|+fbSm3K^+1v9Cst_$$u<)8dTGf#(OJ@dVmPO@}Q1E|$M__MFui0!WimVd)r7HYo!TVdgkW15fi=i*S$XN!(FVxe*xVr|Njk^OA6{L8&kqVPI zcXlJZP>4nPcz*Hp11%x@ypxiS-3LB?;9mf)8rTn5U@;WxdVYLL=+F5%{3JwkD^Hd3 z!F&@6(buUO$aWhoYe_#_gixCQ3RTDhF*@RS5QgeScx+El)zH353`s1p_VkJ~MAx8$j^8#;ld!g<@g-GB` zM333m9W{EXU1?Q5$=})#MTl2SxK^rt-R-}fsw#w(>#^m%@+Ex@Pl9Gs=Y%~QS}FgA zWnttL4iV1MD;1lz<1K=Jfw)m`vV5Fn4@D_JzQylOy4;B3x4>if+HVE9!6M*n=v6}9 z)LZua{kFV9C?^UQks_q#G6(F~&5gi(2rCYbsf+q&ihQg>G+u%ApPAZ&JxFqgL&+Uf z@)m}I09&@>y=h0Lh68ZsG$5voF(ceIan0ul z&#IV~mB`#4Ztgv78t%pNKLjVtL&AI-7Vjnb7@>ARSl5ZX`)qoPEs#10ztP8Np-$rM z<%Ro7ERpKUSd9Hx(?x9!DQB{m{$Fbk`Cln4vd8X_X_Px;RRTCXr4+`g9e8zz*MQx3 z_De_FGm?-?Ia2rW`zH20{kN#4L|N}FD>eKQlmL6vRWojm%qF@*En&h!#G5B9GcjkTAXmR}4H;>AN5Y)Z{=w{pGsCZ z&zpiosbO?__(_hehmd@{?Sb@876Qhsr;1U5Xa!Ij1=4RZ=)8xCyQLCttgicK&gs`3 z9M3apvVL+q&u%txE~yfVR1VNHx!fu=y}Gw#MaecM>nKSS0O=r6V%0{%d+`|_lA8l=%SuZMu5q}MaQ3^dusO#ZjqD+vO?vnrxg zy%;0CX@VJh)*{3I;{p8<(`IT>WXwC(5&Wr#2VAhD{Zw08kW1WoBe~PPn1SteJrq#m z2{Pj`Ro*%w#34?EGXHAFVoso(G=gkb=Q|xw2I1YHGL#SHo!*`zGb<`4(dh$yrWsjd z)G|AcC}{&yD)&e|@%OG~eS*=s$npq?$#fQl0fR5d?iSJwfqbGRLY_V0Hus79RlYU@ zcLK-eQ(>{&|RY+W=V;@{ruwWW63+2O}t4fAU137WxTvF8FM4eaTQ~hYbUBU8b;IY301#;s##OyO%lLNDVNMud{ z39jDTM@ZsUU(tf3*+;RFhLzcG_(0{}4 zw=u=Jwt(-Us;=8fq)I9L6_y&`Q4380Gk>2~E=kW<>Zh{UtFJo?Yc zHsU!_E|v`fDKfKWN0u}Bo?LDFZXx?Xgw0e9F-|Hp*vzX^)ko#Z9VWktcj$V!Il6+{ zY3-W>PCwzOjvn~e63l4=l&|S^+7RVYXD7O?=7lK&xu`#dmXoKRxILS7f9_h{k(~Te zU9-^Y2~=OujvFaq(+uh1qxOEdA~hE8W&);n9k=!iPyR9tR~8+cI9TFVrS{=3+=z zz1jP;Pq8d#&dV%7_yx+R^LvPknf5#9by88I?*1QTVRQbv=I*o;j#cF;MG<~)@hk-? zU9EYeUv4Z>`qL_>ty@s|x?OLuQXWjBWp4<(H<(_!en>pg{N`$`iCLwiVP17FPnH$S zJD<8Wx#kO_qGllr$8X}lq|w1HI)aUEFGKW%RqTs-yB4HA3{>dr zQDPodz4m>eAcp^xurB7}i2AOky6CYI7E#=)^q&+(*{36gU(OKMtKrak6r@KUC@H9Q zv)3yz8_(p3W)|O&&bum}PXFR|JhOri@r%Ps%qW^WX)2*p5As7tdHM}!t3S7vatHM_ zo{fkaVqA*uWD6SN{2u$O%F)u&UGRIKa z(MAOQXeoV0n=8+wM{mTRAXxV+?JnfCVq!B?FZFb$Grkv`I?(slN-?dUx6kZW+Lt6v zt-u#txOm57bacMIROS{{1w$1aEjkorpGj)?x#avb=s2f$tk|cH@j;0pDLfG7G$A6_ zzIQ*_N5S#P=uSN%c>RqrKuZ%4F`SyU#3k%Ws!KcW)XUs?Fo^ z4MHZ2)1Pd-9wv>j??Q+UZZ7A-L<{qxz@Xb>Gy$dE#X z5HT>?{q;4x8hQ?$RL9x!-?g1zqgBy-3Qn1wt1j7{WYOEBxiO^e9OP2|N&=oQmqwx{ zQxp91p-Z6$T0t(QI0K=(7Yg4QTH!rsVDOO_W+SLeGp~I`_t*V^lXaOyX>HQVqX5bk zO~79ja|{#Jb(s6{X#qTS7YNP|$$vY#{o>Y+zhmEA2suu&ardo1n(5&oW=pD9vk;Pn z0|c0waiVUNFQGE>+@;dIJO3@O4l@56U$4TLE>1t`Q+Sv1b?2&>J=kZL^6KDD3h;Tt;?e0Xcn=+QM7LMXaOs0p@55#E=Q_&#M~A0W{PWf)0; zBB$mm^PFb9IP0EFcZ{rBtZwGCSu|hT~t-sntQ6t+W5HOvs!6ZieS7K5oXf0eM ziCB#q!@trzeK$zsQ`@71E#3N|CuSc35o-veD9?M55ohfhzDb1~;R__Yk3vAmDvu5h z4HosOOEfn&&V_!(mqwKx-aG4Th!vi{^l9G3%?aGN?l2p?LeIIn4W#_o-R=F%rWexn=xBaN~S8EPSQKmLt^)=XC10d6aRqzTfQ~U^^#I$9w zvZ!iG;`W%yq0+EYb!qG=mDe@?`iZd?I+&Z~5r9q*uBi2D-R9hWv*t^%LnhGZjK4;h zTyeub&6wkh@66EC3EegR@&Y-g!1t8cfMwN^VJ}E?w}y^(=ESwP-KRV(+Ym)UOs%8` zc-B-G)MU+03tSmcWo~Y3y94-C!Jj{WSTjXb4=E${+Ky5?ql}CU((}Ff^$n*|lpa)$ z{UCV;gaw`63a0`-etz571KlAL#Rq{ALwGU@@CHMEVtS^J467f~&jTbdw&b9t} zneKwI^+$%oN#%?0L>IUnvD}t@FKbZNgV7ioM*6M&zn@U4+=GqHLvw$=f$H`QZP;Ke zhWQd6V2d*JFdSoHkL1%mTrYYp%PmOK>lnAj4%b4of=l$ca-uFRb52BVb=MZMsvi zN)fScZaH`@FsCEWVkYa7+ZMDN@Yk)_RTaB-rU`kSA!cL+BJ{Itp#8v62=8t>PPg&6 z&IZK|)d{9CS6tt`0k)Y4UfOK$T4A%mwTD|PV`DtXBaxSe4cB?(2b@fBi2Xl-k0f_) z(QiFAH3pU&kDLL4^?SR;wt2m1e6GR! zZ3BR;^Vi6JKhASU;+E9)?Cqz(>?h=v8y2@Rm#S7x%AeKJbUQ#v zR_SX^54Z0Nk8aG+{1rW%6sFU|`UWRZw(R))q==UxH`(@EF_*$yMgQ=X9j!}sy%Wgx z92LG36Kw2SKg=WVV93-g1om`8vGrTq!>k=Y>vZ7vB}r6>6Dt4MUr&32{D-yw?2m-! z#X#mw+j~{7{SfIv;d)aw&Pi+jJILz&^mgit5(?4x1b- zD+YnrEPZmPjkczmxpI3N(b=RIu=ti~YN?H_y77lX9?Hpaalop2b>azAZ0e2wSRcak zZoF`!!+PjrF|reXEg5oRdGvH>@K?wdc9z$V1I)}3eJ}FY@F&ct=~`i5EBwM0eiIsX@rGiGNW}_Y;PbXBn^(NVT5f!$8TP5 z>O`|+D)Wy>kg0%4Dz;@ucIMVxV{#EY2HlUnCt+I%1N(Wh;@8Il=;`f?jEeFu>B6Q- zhqC9L@w@OEOxvZM}!PS>;_sp`W!uLugBf&Xvh5| zMk~aiggn4P;Q9!mzkbezUDRN#!wfwgu@N5lbK<5SPYE?Ff0RHT@&zaIM$gWF%oST5 zm3IKLW)E_3%;%+#j)-G$=+t%utrxS)-2Z`}hXrV&F0w4*?Cid2nu27qHY! zk#lF%#`DRyU(w%Z&!Ec4i`x9{SB94-Bn5-2(g7`;1<=dc3k4AuB&mY#LkEPW-&-GW zW)jH&1qSY&c}b`tryA>gC#dRAdrlANUVOtBLbQuOmqtL%{$QvYdTHD0+R{i&u6?)= zB1n~w+P=RhCWQ1pO=56D*}aU6MYzRCop&3rJ<&RJH<}v7*oPwI+sI(`F*qw{G{CF} zoVN&+j*zmC)kJa$PL{Tm%G((plrmdlz+V1X|O&Uw#sD zfd)s@{13jqr9X4cg;57=-`(bR>?oFc8X0B*Z|GC5Fz4l3Ql~T( zUw%du(?!gcPxD*mb5mEknR-vy^nVue9M1S_1N`{xi?=JZzlz{FMvo^Avf|gHVk!o! zJG|T1U(tL_F|dC;`n!dU)6Dd;#d$@weboE~o}!&|k7t*d>_!v#QVJHUlh^Zo2OR50 z4hYLTfQGZ1x=-kZq)U6Seu!JF7)#GE2P85%yZxAeEx=D%oEPrSHiri~Q=gXwI4st6 zRx(P|tqcy`YOpAy+$;BI&!Y0Ina8oF6SOr&gAa$u*=zFVb!pkx%mSL&R%R91`h=>* z#&wpzuJ_*&Tj`~anh=?jbgMPu;_BN+8g3W-oTJ*@c|n)f-z$?8UE|KS$CT*K=qcJY zJ0g{|Ayt2aB&V@*5pW@_(h$}pO})&T>Vs|%pZk99^Kn2WFCY931a(tr_%q6|wE+HT z)~mCi2pdhWMhO$u;+-{yu=T$8CAw#*3`?ZpC)cK()s=| zCM)}+RZyg^Xy%csn#F7G? z_XW|WtLfux+SP&N5=WX#uXC)wa9xw+uv7W`u5z6_hN*w?=*J+%e%WAB0zyr~%cHu` zS0Ud8=Q=4VsAw~1^cDxk&JW0!d6!s*4yGW;?3=@4=1AZ1xMdY^hFY)CUfJ<5xEqM| zV>c=G-Z3=_Q>ggD#0hBfgkq=`QPF?gx9^+TC;CQ%#z9AtG%XT^_3?iOod~T`TZ7xn zkS4uw$(msfY6UdJgG}i_AJ(`i1jdm@4(CPE~$tTh(G2!%EF-ev)OoAfGgS zJ^AiUTzveie)FKIffFT}Z;!Y2yTFnI%2AV4=1F_Rz_IG+LF*#Fb@P|#@uLuaA~x3I zH?;WYU-I6UbmTVF=%(n;TI{3RaXGA%Ki?&);CLu9*%NG7GmO<0cLQ%MJ?VWO#-uuD z6}AQdO>2FZQy1Sqr|*37;qIr&-KXy#k(0yLS^RMD94B&5t&u!8zV<#{Zs zcRPvt=n8fq8<+zO-vQ(*-V4BvTtXP8M#yP4!mzR|fT)0kR8b->4tzaI+g=<4IUnE* zVULNnM*+YR43c6ZC;K2`I!NOTl45vGHlA;~>uAt=aW^!;?vCf7PderdVyyXbeX5CH9%_wIZ8czb_~%fk{%D3NET*d5JVghCHOEFe~& zfO;(%oAB)JkQh@mUkE9C;y3Mnh{+2W-G+ zzap^Pw?j7+o?>TNp#_kIqF;#_ zrm||ZCy?$Z>9aD4(6VIr-FZMy3@_p3QX9GC^k|#9qYD7bbU}kzj0r{Zj!P#VI*qeo zJrb>At6fvc&Gz>Y1ZQ9_c?Uxdon@y{cGslzkAmg))Y<>;?km3_o5 zHw?9hth~H{U%}7ucB!a8+077pWMpNL@#*qp9K?={7$R+2^Ap_#Jgnjzu(PqOR5xB) z$CDlBz3^T05f9@qv{ssF-LzDYqnDv!d z(cN%BrE=JL`}vOXwmnH@_p4WC8@>id-rO}nD5C#Tc>Etc*>x%uv_Gt+@#;EGwMc&> z9MR_dg3kch9}}K?59Q{#4zmHB^3-)}kWq@NqbUXdLN9lDX8VHJS(<-V;2Q{zD^!_9 zwjazB6HFx37C77F;r_PH@5DGN_|2okC9U#e{X~q8$PR=Sd_ANS8;Z%-v)7K@i}`EU z9^bZL{6A2kI{)TPiq!Sx3`tY<0yf_kEowXo96%?c2Oz}H>Fx++Eyr!r%mGm@0qvzx zR0JF^Xp&)Y?=KI2157h}CRge10=|F7-%(o1{+A`hj6i?o=D?)ANNZRG6JtKwk_42K z&3SI*JA8gu+gq4*@rIO6DOY5qjCXDoO zg_$V+wRfl$02%U?PMdp6XP1Q}W!YQia0`Y%fO6W;0!_R|oq_pTd5lmQw@`lvrx@@Z zu+Qq_X>{p{Tt1Zh`$rI4iC-IF^Q>x^?_4547ll+^iY|rc|2tJdu!x|zmBfr*dH30% z8lf?{0Yrc#i(`3l`qayVRJ$hWDHm+<93f|~-k)!;Gj@wxrYf(*)X?mHL6c<2ZSc6S z;8HD@`=|ceWzcykLArvepY;`*1aW8<*29Tfhg@K)CXmk6$jA!)_^bmByEr_w14B6S zo`|Iq5C(U)dy+dT9~5M$kx@|XvKg$~z)V18Y60o_W%aPI$kK+cWyC82-2Ki9!tQJeX`V@CUb-{g4R(!BYInzhyj~Y~<&u;~0ix%2;@%hR z)K~ezU6dc^O4!z}rLX*_zd9-^a^jme7i-)B1*KhckWZtRTc1DRMtySIN)yXt`m0yz z#`=DD-J!GVWcg|!C%3zWi3?9-{PW3<8}2+~J0lL0RF|aUY&vr>quO@AqjHR?9je zaiiM3vc0YC1~f{sCCjOtJ%*AUxM8)0@fuSTzG52qEX(}x3M{Xi+^R5aKc$PU%TNV`+o%rx{o=pK)TH39aqhkSGe3>;piVQGF zxLKm@dR6I*!o_mB?uk%dmrkRGq3;C#`@m z*q^T%`e6IVkhe47FvCAjKMF5%69b4=;QwXC+#-gJeY4Ow0AZ13A4F&@Y2)9(GqNbeTt3r)Xl~_H2NS1a#AV+U2u! z+c*MQK;1@bmiNQTtHbVk2+m1o#h7=F>#ysn)tm}a0x&sv#l%31<-o2ZP_V@+)Fs}8 zGiV`VXUpP2#LP+toEs2Sh5tFp_8mcQn*}1f;nL5{eA}j)++d@CFuoy|Rz|92>kP;I z_V)q?!R5r0v%9$QH#d#d&~s(MPP&|bLS#{-D1=bl%%FgRbWyv{XXth#K@3ny3i=`R zKxD3ZpcEV>-+g2M_UguW35FWGR zU`Cb$l1?MkU4E$kBNexX4~Z*y;b{}IDdC!L3G$rP`O3GU((6nRQ$2@1?TK+B4Zajk8Y+pFz;p$7LAfOLzc@Yls+*glDFu@K3wTUpWQ!HHd6T2c>p_{Uju3nne!nG$ZJ)_E=YwTU&Jx`O04(tQROd#Ap42cMyA zxYs{=#9fuNWrV%F=7G&+Gj%#pnWt%$ zDO)5K;-decJ*(Maa(G(zBG5IHDz8R7$H<}F_0n2CUKc!F$X;R5aGqxAhtu~p4%w*{ zIJjzLg-4HrkcXIA0(bqt^{C~!p59Oo$0xF@%OCiMXVN(JoVsrfny-8fGw5>e4O-oE zt-sDU`IYP46@hGx)&;3X-(b(|yu0R~T200+-xtSUOLVwRK$i~Vu&Y4GBf08&O@!GB{;^e4WHq*xOEPuOS zkm!}<{M!V){#3H)0bcvDw)bP>uof)>yIWDtRkG=`Uhx=8mV&w*!{ zM?}=A$ESQH1!s#DpV&b5i3$NE%UI6oU7fC>v;W2`coE$$Fl4M-KcwT8hK$5r!u)b+ zS>a(-as}-wu)x`B(pBqg%L_ibE3exOiYs{$Oa+Jp{V&g$hNN{`RV^iJ$!d2GxGie; z{_qtZN+}NAIJ#KY&QJ7@F;XXI`H?om{g<=5pjc3mBBHY z&T6WAvwtdfFeY;5N?>nTh1AEX{b^A|?asz+5(FZ8aDvY?SleyBeF0#dF7@g^s#=bvLpw~NvMcF-s&v;dFChV}WprfT5s3P_hm7owB8 zkY2#fO07Xj)8lyHHs+V{N~>szVMb%)7N;_^^^(?^6(hRXw@swKk0&bRSeVi%$+V#m zn58e4=|R#b*?o3$MujR=y#3+=2{fI2vkXn(>>OK@WzFf%B+aUxPl1-P*!OAC>q{9L13z}0g4WGtC(!@rp##!B5sr3Be-6a`fSqW zNd#?SBP$Uh%jBdHN?n#zXI~cGOG9ER>EplKVn&bLX_~*LaJns;o%JRzE$uz@2pa)Z zdPGM#UELUgN;Nki>g??1DEf6)s^*^}zP`Q;BJxWdtXIYQ%ISVbhfaM_EoNiQFM~RG z@K?U~heY!Y(%}fMM5trYAO-Vx`?SfiW0g`why6|O>BdvwD@g1qS zKW!<-vJoUHfyw)cxEe10EWPzeUQ3qUP`m7IcUcO`D>fU`x8p{RprTJuk$34uT+2n`gC$P8#NEoEHw)*spf@031X3 zi$C6SOe5(sq3GUO88eO^r3=;CO64DX_l`-bFJ>;BBR#DLRHF+Iz+ikrwpbb|(ODHc78r}~XmrtYOWEk?uL}q&jD0cg$uBI_p2qG zu;*0AdVbMOuK8XU|NV{tqXDE=Z}eUF4M~0}xOrFzzX6MpKg5{gc-E9+C{hnz3B9-}Nb#sgCjRaHP+BTwCc=u& zm^aGG3g^!AeMz(T3YGrgv$(yv_n;<`7l{^ib6zejAcU1Gjb`?w<^90MhiyvK4tvFH zLZY+5rU}N>5RvJ+cC?l8C`(KApSUO}5-_XtD?cVwwnlj=X`je>A|7S$*oaW&*xias z3o`C%LI>xs?7^}Vd7B=FG;DLJ(k;JS6@`3KCG=_2`1kv-Ltw*%&JX zM_N618IfE7i@J2_BhF+1KYZ+W<_x|nX7O zD@R5~W>!@V2suuf10}v{D?y~Zrty9+MEdo*X!P%Q|HTnRR~|MN;jYh`r>wYw)G{O} zB*^|GRmrpKc)?qGo;TLyCy;MH!}lX1Ej#T}Q>8->+p8MVZP$a@uz{_|W(hyBYgE)f zJapn1Z-bo3yvq2egh&0>Z^-z8l3EV6UJDRZ$V0mOK!()}bQR>kILn9wnQy!^rv!8^ zELW)KPpCc|Zapjn17;^SR>}g{@l=FtuKy@hPtfLpTxcw| z!N7R*s3G?+GH1gBeh)SHXIW!0c6J3>I+Z=3#I8}{n~!ArdqCV`JJlDghH6iG?GT)D z5ccx1c9CfasS>N+5~%0*fYcUJRC`})ow>0BVlbGWIjowl#e{6yIJ-EX49gPbTW~q3 zM?Fl1_B}$`x-jn_@)V_k7bu{kWw@Bbn+NHa9R1 z?PJoFhyiUO4OaZpT68Ez>_bHDLgk%|m@?e{_V~H{U}Vlxow03ujK-D;6{I>H+#G)l zRF>0mvgSQoq%R^eQeczkh%3H{VQa!!PF!j|&|f5yAZc6cv<U7$2%gk-w+n9C1hUmR%eFHh}LEy{b=Uzs8GDkJlAf3M5v zeuhDgOL*P`_9TnmA%(>HWT}D{7P-E{dsha0NG+r}3 zplJpAM7ki|Y@iIed< zl#-8ASCGj9X&7oW)M$lgXoDI0#k#V+#{P zmMcP`^^C|X9V4N-OpTT}N}n92YKm4a-Cs0!@Fu6%r6$Aa@+8sE*QQn4;ddjq0);4MqAV;tOO}};9jW>14FB2?^vw&ZH6{k^yIZoK4U6d-(x21iF+uUbap6@0aC=Ey)#AliW9c@k?cH3v(E z`6JrWGC~%ggF4?nz!>(9_GIfgL++iI66SoRlKO(p-UQU%Ai(`qk}#6q!S9aSJsU-+ zm+5c0oRYo+{g}xwuj|ztc78L%_dd#!L@;lXF6TAvPfIL)Ui}1A8yDpQ+fffDnpxEc zMa5?&H0r&R%6@S^LL0c^Ra(#RukkFVPKl`L6fsF?=qzV+C0Urs;M+V?@HA~Lu!t!O z=MI_kiz2h zbeehTX0kG$MjUH|b;O!bILt$MF-{^P_Rj!UoA~-wx+cWh87pWj?*;c~rmx(H_;C8u zNSX_{HJ|-b6_5vHwab-TQxy{sX~eW3C&c?emc&b_=@s$>XMizlC=`C`g8g_yI=6L2 z&xqIb3WyRQrUrwS#f}l zpvY1|8oZy9=3P19-qlv14eDmk9;ib+5AH4k_aSs^%kc9iEoV0vz?RXiK*4Ki_vhFw zt&l(5lg05y;p}1f0rKGWN@!->f4o^)*^`zsc-3DM@uX|U#ccXYhLTOUVe8Q&j|**- zc@$@^bpfYc?S*Ns#^!IIvcq`Ldlv|9U25ZcsdtbcAR3w?z_SmT5bxi-kFl<2N9ZJ7 z9t1Bt_#cc08iwEYxyV8;q{wZIa?$_Y8;n+#7illMJ4%`B#JCAMD!hU*$@~%{-M%&0 znS=O6Jk`7ze7NeEf?8ia3@juq^YUzKUgJ=lolwtk9p+r-TmXwPf? z{{rW-vc;DDLK!(Z&#z&buND%<#!ca=JjPEMIuj;#RfWAw^X;8B;ah#Ep{=c5qa zBxD?+4qLhZib9AGJtrLd0_+HGY}YB32yKlmM%V9CE|!J9YpcAOX0V!N^?NseL|m=7 zE@XKI>@)@77_L?q!#PXH)ByLySl_&XqaiN8$1&E9gHyXfy4`AxS6#ZgC~01_rtz>| zsQxO;4x@C#X;&b1@&rF2Jcu(2b)K!h{`};T=hwDqVa%l%X|wlar`97ugsXc@^xmt9 zNu!&2-#9KwtqUruX=*Vt-`lOH!}UHPZ4_zT%e=9TA?i9N`%k%AYm$|gJA!@QoEKts zoNq_ylRS>f84eSRqnyH}yLdBfzzfLA1~uBxQI@C~U>cNNH~sXo+%cVadUI}>y)cLL zwHT$Ym^Ba*{e{fKrnrad>o338VV-u72!3&Ns*s0;$8nJ}=!ELN+MCcv*D*D)G|(^n zNchS5VuPc{$xqV+l)F57ED4n4zJ6if`y`Uce@VQ<5JPbEK=9nuzL`pr*<+X9+vW}DUcCp>)9$+UBE0NFKY-}a!f_mh?b=m` zg(uDl$T0}ClDsh5qQp{A8z1fE>7cCOwL0GLmdt6cZ1`1IC`F$m^@jV{Sp9pjJP4Iw z2@s=Kh~mQDeFv=tK6XPmHVF-M#nn*Q1Wg83fX4x1g$#tQajPQ&)siX`aPCANC?S+AKtb~@!W@NjprmjAKka>GR8(!) zt|Frd1}dQ-fn?=rZ2oA>@@zQ~$W)iA8nRQVDeVEKvijoF!dPj^ygk{12c1j?tVfL~O=UqhH zEb9#Pde`TpwkInqjC7ys?9NttTr^H_w`eiZiqT%H1MWf(;Sl-8DQrzqdkb{{2%#-DyF@$m&k=b5=hP_}=WPqzia+P!&LoXqL3+fTTfZv3$+H=r z1{A6>I(0TSzE&ux(Ttap}s23?b)ee{v{K5#cZ^gFa7c;+^nvPvJ0Av7cbVI>H*tX6kf1RSSr>(7irJJuX zHp@}dnD@)WruJ!>%$saXC2TB#xs%q;wkDm{0$p>N_Ssu8&1vP7w9$rqe6bs49UVm^ zQ$GVO08N?eICw=7vo#ClL3Kv^a?7_TW15%={s0#di$K^Z_SZUkmMD#qrZq?<6^Cv&fmuoEx_@1z5MS2yOQ)QB_R zV@x+0pdSkKFI~Wv8>EUyl;XSrV7Q=WBp~%oyNt5$TZIN$ODg}$JA^quw$T?a9vJ@K z&91fRa@4Y(XrZ=dErD9t^6x0llxA^(r{bf+?^a?=Tbq9*yCyIokQ5^eR%Hnr`QL@Tq-_Fo6)q_G2;L%1Bu*#)8h@TqF!VBGGp($r5UH6@;^9dAiV(`|@+!$9h~leB-6+>K%;NhHw>z;FP; z1hX{iVDQf=v02Qb+~_~TWu)9>B`r!mK)t(36IvfctKM-Sl@Z)KA)Tv77{1EI-D1FOwagSlt3DWA~`Wfd%|=6 z#J=5f11WDMucs91(QOW~I_hWR!gFE~s^)SH`NNStfTMj8LfNeL`$YdJRnZ%`NHr44 zN}!1wtm#e9wm0k`B!>38Liumh3`mt!WynN+kNbm1qL-2r-^bZK!8L8t`0hM90Vz-z zI+UEeD))z~@^^^I_aITFo}9FnQk{SKh{@vjRWIPrALWca0pYjE>CQnyQ8t$w^37 zpeDq2&aIC5P_0V-d}SQEE^}(Hj%(K7%!!;o)g&JOO}4a^f;ypB-T^5r)4KY-UHEJC zBre*d+~4Nbg_g^wQsZ*8JEj!zcUi!IH7z?J^q>AW-d*H~z3xY_`xxLS5oSu4`rXfe zBkbr{@R@UUo5D$d-y)n8!oW;9hkrSAFEgMfgu~=$ zW5<<$*Sd(5*o6amcY}FnS-i&1ETZv!rxq11)E6#N`R(Bj2vwYURaIN<*2op}w7oh$ z4d~!I>}%u@(nZ5P;{5dc2fDc7%miQ9n5b&4V6@u^j50}c$&^ zH8bH&G^A$3JpTY z*}x_aG^U`l!bSD2xpSt4=ps0rLR6kE(*vVE@!a|edzfeJ3~ZP0?Ze>+Wrr#b;ndaD z<^Ew?>IRBS+rvbR=KTl5`Taj&IQXQ$(GpyaMw(sNc0F}>^@5zN1}V#ir>su;Hdgff zj$yWv8!DtgOhoJX!4Nzma4C>pirDSbXn#crTb6om+#_>6k6aeZ4m@?bMeA{;lsz z$V|V)_4Z%g|C=yX4P-3gSnZ7w%D%PM8HVYGxSD*{&wF_kp*~E9?p{7Z!z%$AvCw!i76o;sDIT0_B6 zscq(s_mIZ$Hw1wUYE=AdkG%vq8w{q4s*Ku&h&5D`fp#^-0?(^^nQCb7c_qsTrZL+J zjJ0ggL)&CBye5|+aecX~;mYmq`_>$FQpPAY;$w@lsY4wlG1Li<`cLjIHfcQ=xGM>b z+`KptOI`|KBYo&_1@}nHZOL5uk33s&`4mkgY_w1%C zx(^d4L-PB1@K&}`tIJ`w7JNB+eOp!9-=urHG?fBuf_v#45<(`+eL6|A)(|PIDpY&h zOT5fgC!Lw?=tISkr+0Q|AC)wGiHcw!BofN>a2e)?G@KAL7FMk}Ev-lB&nng38;L#+ zcV^swbze&6oQm=!ESDfZDd#$$_ZB0h@+WDAk$FVI?l3xaMr_}ze~II8H8U+JB#C?R z&GSCtW?RXR^~dIB=cql-;eO@#dl|YTaFwAC!E~2*pQw}I5CNUl*pX``orf)GPu}zh z{a8|cBw?5L3s^5iYhu<2KmRY6o69Tf7!L8}zp&}dboC-diHO8|9JtWFKp0qxnwpf} zI5M7D#LBSG>}28p6U|pI_oJKdbG)oq4VEgQ51-x2yH)Nv`XLFOM@M7h0}yh}>+I}o z)$XUj*Udb%-_J$I!BEAk8!C5qU*yZcY7q!&pF4lHe4)t4HWYDwK+8kZQ;KX^Axiy_9>QiRPj57W`M8P7~!iV|kq`I3t5ASI$ zjqFS!!gB{*Lf&)`6XEXMqz&p4X`B%xe9BBZoSd9f(}4)`AG%Ic03HMHlg=&LHg7qIgY&xBN?8LD@loG-O!MIDd02@FgGQ4LAz}Yn0Ukm=)1qwZj zb-wQ0wJ`zQ6|_jCl$wkxV-a)TG-#_r-PSL$2Z&S;!=~RO?J1C_(=y;I08CLA)TXNP z1Nl!h?0?VF7jPVYXAKo79FXJu{XCEv+2ez%OyUD}J~W2&1{zeP%)tHtV=^feY3>KeU1)oe50_KEyHTd5OyF#O(0)l| zlegvjdU$I^Dt|`alj8xUXD;)KRJ9(SAyee8y@HW`u+EpK(=Tw6@4bQ>XG&yFm}R86 zM2fL>L`bWocZyUKEk+`IN1nHkP4E(f~NH z`=(n}#5tR`vB4tVm*noLU)QzGTm4292IsdE7LP}dY%3SkWn^hos@zAK;(79tWuW}(bc&*vifRKF{)&fTlpRG{c zZNcWZuh&qr2Wzcl37b!9&>Z1XzjI?_fQSA_ss%}p*>c7iarsW`rtNSs`PUFeN-yCg zMX_ni#6aSx3XRPvKp7r-5c{?0dzET#x9XLkLRXgQ}_>tEe6cjB(5I+fRSF>iv z)f&yt3$~M5(2xv9n&j}xEXaASu2Py2+fm8wC*2d$IXU5YiV9VHVq@#VduUKwTYf!M zsA7IjlHegtPVF(Y>z_S3ke)DtN<@@F(2*J*48$XMOJg^HBB_-95k*gzq}yC z#islzB;W--e!~(S^A^|yl9n^(VdHO`BrWH-grKpekF5FJ@0AkBY1W7XZgn4>pC2A6 z4*dpcOu?^SeEbXGV8ABfOPb|*2+f=~B~BqnGZSt#&%IEQP34_K<=>A)pGR%<$8S%z#I%HjK*RnO_tJMx zkEFM2bkxSC6S#lcj6}EKSYr$5*p(Ie8xRTRLrD0;ammb-(IYh9*_X2WipRO>Eh=v0 zZl1$|eG~87`2HOaw%AidyHP4bI-K@nl9Gh%F6*x^L6)unX;GqOk6o_kgxac;Cc z5{pnbSnQ8QkW*KQDhdfV<#LuZM$VT@;D-gVo~K4~5*Nr6|3jet^ZcDRPCS z?1Au`DQ0Hf=)| zbkG59+t}K>poc|(4X|1_+9Uq#4eJY$-chkN%aguSwxS^g{)niMG$F(>C@pHa|Bill zbl()6rh7ZYM67R^9{2^X@SU1)HA0_$Ow_*-_{QxV+>no;(S~*c!pnt*vN9EyVM2wA zE*nBQ_6R?~On*5~oNiBiV(+9z?xuwrlE1Q5bOWZFC)4HgB?U>|EW!hQC;v=Wg2lnErYYgBGjo5!_^P zr+G!pl&Qf^Pgz=hZzERIN6QXNRN}YL2s&kTE?Lcv*JLX2PKfoLu9A(P(HeIb96I`> zH@Vne%=^!C8%>0T>4Tre|Gz$RJaeSr>As?T10M(JquPVC2#Q_|Ks3ynrHC}lq4~B2 zgd_#LEk7>J=D&(F_ufKb+s+w@1ABpRxozAz5BU9Vwgi)>Nl7Jx>=^4_rF22tDx8tz z>YioSR2AE(N5&LNVarG_9m>^ROg|CYn{Ip%ZD7WyiHtH6Xg$>=;ZWQ#eQGy!TRoRR zH#-Uw`6H7kptGp5i3h30fskBidfuONGM;o)e(e(Aspigv@Y&|}0GsrsmnY)CurjOE zJ|f^~UUW_aSnNxS8IC;`efG<*Y7Djj#@oGtxS)>kFW)EvJ(1Hj0?-q%04rA&IXVBg zTo#kc2qU>?0>n0}mWui|Su!WVv{f6in3)0pd*hSb)}Q3s0KPTml^q zRGjiAfgzc-BQ=))6F=|1djAEGoLB-_cje(@X9>|a%w_CbRdiJ#`l`u>{{2!;hJBqR z7*Q^32z{sA9KDN-vS69#Fs)Ri5un>LZ2Nj=Zx~<`KlVG#wPs2&6gxRGt*SO{QQR<# z>#Bl+<|3{8Tz7Bgrve6|;cwb=9Ssg#v#UF?2mkC#sTb(VCH|^To^()ZKDrK(%3Hdq z`iQ=bVeTpDb(_b<@L$H4zIx1Q3-U!@6bsRI05a}9V zX2l&R=VEHUl$K!mtGHse#WO3>bYt)xbWMR7mRg>x>9DF@)bB!I^zHRU3-FwzfzOH9 zu3Li<9YP(YuD=KngAYZmh#>L0b6lDsQ{Y4$ci+buyOEnLf=SB=#W})ZZOrBmqm?e+ z$t2vrZV+sD7E9Uv2|e6CX>QF0uf(`tAX_fQmX}b;b{i(0Dnl6I9IW54CKq!|Y-{d^ zUZdUufuEbxn>^&pIG!vs@Uy)`B@UZw08ylz&YjuBA$J6?tL_a@9nN3X5M<=HobJvK zlDlyCg7xE#_X3;xv1$POzP|w?2Hs_}mEA!eIks;vMXu>-?0JZhANm zV`p%X^#fTHmofV1N>k?2*7{5lW>wDWAcZ)ohLu+Lx-jW-X>kQYzOyv4)~p@m(R~rx z$wP8o$(5cWtr!HG2e-;F4s(ZTPL@0|w)e@gR2!RbAb7wR6?yl-feXx{w}s^M8+0ND z7pyIzvvs9<)G-0E@=QO$!@W9ES}t5n_tJ@f!dzl=bH>qhadlu3t)I*;5TBPTMbGm6~i9NIA* zxt8AX^M2z`fDT!OiHPUTzhhwPdll)se7i5mR&MmrJE;c1m;t9A=0@Z>p{+*SY-YTC z0if3fe00;kl}c_2W|ERmQkVLmM^}kRFNLAQ?Z(74vqIT*J3;fQ+m2OQ4_yDaSgl?w z^RHG61#3>DF^MhyXqn{r1k2CANMsCe=0q>BR3fU^OGri?c#HW~z{k2tBq7Ue?SwLIOFt5Ub!`c*j zeq4D4>J2@!3!quRm>EerYTmqH=xETyg~btESFidF`eLI#Nx+`}&b$UT#`%LkK@{L> z4*qq3TB%#olwnL&tKOHTL1s|U!iIQKIg&Jp5F)r�Yq6#Br+%5v_nhb=9tako?F$ z%Rk(+I@12gHwS_GM7-s&d*L_S1H{Ye;`sUCdiw2+5rISEJ*~)+f1P@X1^{5W)6cFH zTr-eKns0)_MGqS%W?bHqNk6m?IFJ_Kse|)Le74x)2QSMROgu|185E0h>v7nv!%vt4UHX6`(u&WLcLOl-#)rZzu}*Wdh(l5RLo!N@~7W(6-& z@?oye<(zzZ-%}BW7l86z&buw5x~}r|o}S0SMZq5hk06UeM80ab;1-Kox?syMkmZv% z_rYR<=K9IL2ag~Gak?}($(q)(7kP0xEM65>k9#Q_G;ydijfua;xiwhjYboSHx%Jz9 z`qDSUqhqBDhI}YGPEAtIDQe8oA6DF;JLIk^{|JxaLma_*fvKz-7UxF<$;F-NW$9(b zg_Viovrt)j(3_FJ^*B&hRg2p@o$ce zGo(*j7vgDxZK6Qv$myeY8-tXV2gL>26Ir#_V&hDF*J2aw<1R7gNjrb9tVYa zqfo-LJGS(T%Kd6Bk;>Y2N`lf+jN?3;b*2tAD4LezN}_|@6WR9!ipQSa6{54}((#uS zs4}~e9#B?za0cpfimIv;m9Ll3Oi25y+m{tzdh~q7N?pg~%b(1EfLhL<_pg`P@+~0y zIb1=i$$<*t*zhh~-vYFZ7?IezLX^Vzc!uF63f+!4pYXf@fhr}g>H(H_lw-tKu(Qz* z-nBJ<-Snk36*+#yyms_#lkchlzy{0%vBn#~+6=9^b+Ez6AyKNgVeOPMz4h4(>g;3d zLegR-l}(sDw+C}**}|Da2so(+qtsX^u@nE%r5|`COVai>&6O(J>jCv?yU#gxnoLhLw>G7GK)UtQ8?P0!eZLRLAZjITxB66!ADP*+K zGkBw2C*0M&PbE^0V1*UO*rQfoCAE|Z1ZDZKI`ZnG5;@cbN{^9od}F3uc>ivM*(Zal z<%`sMdlmn1I{U&c_j)$6ye`UHU43hzx{>-B#dDtPPs__b$GP}#ttON_mE5#j2rrD< zG>rLW_;mBYyFq$Rl=`D2>V$N6Y?IuGs5)hmWsrlF;tcU0cK8@{rh@7tnMdgRkI=R- zT;W_jD4{NxA`;KyM;u+towOL|f;%3YFs+ZjMn~_M^IBZeCW=g6c7kwoE`?>(LHv5! zpl0Lx1S4KD!Bv-l(tBf3Ia0-JL1@&uO~qG;^jfTw%~YOzQ$;#=W!QvCoR_9Lt|QR1 zC78$;gsYMYA28s>PpyikU6WHK8M}@(7trS6=M-2ye1i;MNZoT$TQ+U!Ui_KHttHhO z29LN$x4B#9{BiCY=*I>bki}l=3K)aM7?}1zn`xvkU<%O*S&tDBFd8jB6@XP&qhI9p zFEnZ#QO>OjEL=CRkFZ=Alroc*I=nrv7L|tIIB_@7$4-ZKq+!D6DBXkb?l8?Xia_kG zeu!2iA5YJGMW{#)H7clDb`U(PmzE0);;d_jpv2um4c* zTWPB*RLzSb+Nh^NP3dl<=~jKZd4g9*I0gD7+LXgG6uZfS1VxXN63Ht}G-4yAOb%}C zBw$1cF#ktlhqN-znyVaX8JXps9&tZ^mhjD>hiie$fL@f0>>0<*xyq)TV>ZPWel41H zCAgE!=+!4CmrZ<6+VDQcdEW8q;^8ygnD6qky=Q18&DGsnwApVY4w2kjeYZgrKqYcl zkoh)8WTNGK=j| zy!0WAP{yjmobO42_MX<+v5Y~1iL))gao2?JF+@0(Zj2HXxipF4FVI<~28kw#R^6Z5 zl8T7hC{26h{7fNBeyC)rESm9b66JI;c~d^6n0c@&^^BW}wNhbPL*|kXqw#{T56ELqhba!Lr3x#RK|UkL|-?}-k$a?%ixH#v~A;HC`Q*BM2(}F=rF&E<2SCC zy2yLoyf0*gy0kfef3B6&tqA8ur_@@iTo`Da&0SI{ss9+oIh^ZPe|b$ZI!>`@%do_B zf=s+-tguJQLA(Kb7;nq&Pj}Xn_ESheMYfLkMf^nVkV^j_(LIHY3Wiru{@AqJ&LY&_ zoJnJ3;@^u5qrgSh&_wNgo!kM@=XMibZv^;#64pg)Cfc;pGG@)ObxNeFTW4JzxSyg} zFJCUzMDjq@L@0j@geQm1T^@CH~>Nm%ZD612!wy}~aHA;=Uq!?by zIo)>h#C4zP^0C*}+K>-(^`>8+0B|3%` zNjJGRaB?)=i@^KsP@ax_+qL79X@UukN4_8#B(bzQFO5{>rUgHZ^Jh}y-yfZ zdMKbzzV`F|?3Em5)YVObjQRSTtUr204!va55B(LVjgg5zpg*U6LO~}@ilIc%jPZgu zMRK&C%RL8K((5QzgyF(1n#N4bvFu+smmL4v=VH$5+JKGz_m%nbg56T+5y%J49@&ps zDFol6&IxF0ms&dwIcU-l<`891W0ll+r8$6H*G#QhT%76;@wDmnB~dSZr;VDdWLZ4K z#*;@!H|2XX>C&qD@Q#>OTcM1<-(2ITEhq>eD>7MmwX99IPetRmlxSL)xlg0DxX={} zQ!}F!Y~VvsRpK=t7|J{&vUT) zoe{HmC`ES8nsyZ<&D`15ux2O!6<6SDlA*q%N5oQkCRV{7bO)xxlZ+?sK2U!&_D8x_ z8vPdv;Q0Zd01oQahds4DS)w&3F){v0;&0FqqbyGNr|G_p(%x@78-&g%A)t1Q!0Hag zkf6rbhVveZs?#*vVo2LiC51C)G6ACWh0QIS=z~tVh0ok&Y`in$Ov3HgmyC%Y=&_o5 zPbCbhG6!PUvgK$U{0$mX<>yHC$0{m73&oWJVk7+BS?u- zB`&y?vu2`|8Svt^*M;BvoL6Fai?*8R{$)wcVs8Ci;QT=f$1Q96#cYwP`1BfcOH&lX zO~31j0G2rU_BdxO8-rZcyILJv9;NZ{B*Rp@GJ?n16g%=L09)Y>p+IEg z6@W$dU*y5Q0-oS&!BQcy%D7`qkM}D{|10g`?H)A$d&v}l?1XX0WVQbPm-im(|9@cr zEm(fqHl68xZp^TK-N*6dCC@>PLK(7ZPJN;T`p0fzP1={}>95_6@$ge+4=tBB2or6t zI2QAkB_VWc^x?g4XT8~vCzvr_c0Yfl;P!@H>2OC`iLbrNdU$r->Sq0V=vZn=n@wwW zs|~Z`B>y}p0#bz@xTt-NccPcIU5AF!=>*EjUp&*kQkL}*9>9>wc91zm#1ZQ=gf2xSC? z;&7@F&nVzg1V0H9t^iAnPUl1K-5Iy%8w7ga$qK20CEX+>(F4VM*!X;1_yx5xR z(u7d*+${xwf0Ll$M5=E7X6$?*;u1x|r2GJ=Ad9~djqfFJKvxBG(S}6Z$#yomnQ9OK z>)hH}H!T9KOXJD*Oi=c`X=(_rB3!@?lJmV4I(C8c@g~TszigTHeCmi*mETU1PqA2| zeeGs7VJ&>a?AN{8MTiULE~MH7AzEE<$x%LO&R-7T{nMH#bJ=z%P|5CIsay9+3Lb1a zM5&Dd@$+)=@)z)Z7kNPo_&v}n?Rz6_zH5LTgh}+K%{H*LKLX=!tz`5ZwkwxonT!dD z-I9?0Hf(f)>ec6@1IstKz4-_!_jMh=#T+0Q`9&C<@Tb4|4Z-!|WTap0P7(|8m*txx zdOX#UU3I7&{4Uq%$N-5OIZO0&li82Lqb(+B-gmBxb%xWUztla>;X2{^PUdgvE5o7*|gH%E-y+x?iycM z53;bvoCet6gz^LZ*d)142clv6v+_?8ZBJ19Yx1029V3G}`+{ZCa+09@*R4gL!^g>f zUCdl^gxV@UpP`@uGO zksaVaS?!_tt3-aTP0>P!W^SA-HB3S2V&6e4rZl@~y08YwJS2|S$40<0w91CwfM0wu zfgA`}sp-Q-#ZVEQf|Bc!{kUkNpp4fJ2m@yFBYY-OR*gx%Re4dqLm0=|lB2htYn`a2 zjkNkgG?_%#P4E?ai_e&Ng}z)4ZyAJsRxYA+3lrRmrQUKeE2@7+gMPwhQKv#fJ@hdu z(odx*SDpmU#_Xy<%PSivA23X^EizCp_3TA-nC_QowM=j#PiWCIRkN&A9aOEkp7aZv z3T#+|I?Kl3=jTT@3qXu$Xolr{8&{X@@eu$u7@kh!E$T_svHw*^J9uWhoX81I!2QYA#|ixVSLNz2hTEXkxh9N{82H+^^MOjy9B zoh#sGV>ew3Fn;c4Qml`|yu9VNDT$Io8R-ng4?#PW@y{l&a#jNrTNvflKVc102TQb_fLiw>Y-C-?Ry_phkr*d zP+~c3y!eBjRMxy)#8-!X?fU9Vgg4?+EeuBA4C{MtpV9~-c(&{Aa|J(f84W6q_K6>C zI_cw@$9uJ0XUs3EDG#X1>BRR)FoSfzYx$wo*VKez38*d7SVU*fX$X&7@kUBe^P9;~uqJVuSeXEan zLt)``1*^Y1PKV!&tn&xA{s}%;zmz`hETtcdbo*l6+r*7CYT6$0^xIP>-V98Jz2Blu z`sF)x?EMo2k%{!Rvk0a0<3$WOh}J=2a8|1qt#)Ug)S%r`MnT?UdyrhrO#4@IL#lbKq}4driLv~P@dzPD*#8Uu)(a?eD*Wzu&sm zGnD*5R>y`L-xZjhKR>4r6Gi}i3lLR>-E_awmQxGv_IIv38tgNhBDHlS>PNct< zWfXTG=+>bBLb(_se<0yv2YwlX_Y6BEb1`OE@R$fedpRld`FAKKjw5j-0ip`%9D|`0 z+lgLz>>;@ro$mv`2JBp|;K-Z5IG`0pvUD^h#C=4n)&J^wf|2;Hpj7hG_npaUepMe? zyYF64K$$kJYR@&s3W0FslS4TI_G{*dY$R_!_NO3&O_kZnHl_lFh{x+Y#{(bVl8;=X8G@ZZnD-voyU8yR}hSME-<2UMvI$Dg2QJi;xtOD+4)M}byD^35jSGTqc; z>3;yH{*ENt>?onEV)}Aj^S*O-FGHh}9>B$;XY6JiaN8aEzk7hcC92`P2nd|8pWhR+ z{SV?r{`x`TYW!uf{EK@xzW?hC!H|Ls+4S);?>oKzjh|T339vFHr%mwoe`$9d&ehZq pXczQyOYJ)z{#{+r^~}o240qMf>qjsDIspHPib&p07k=RKe*oMoH0b~U literal 0 HcmV?d00001 From fd7a63167e1f2de50b6e13637518834aabc819d8 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 31 Mar 2026 13:17:31 -0400 Subject: [PATCH 15/22] Wordsmithing --- content/blog/2026-03-25-datafusion-53.0.0.md | 22 ++++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 9fe4aa3a..efe25f41 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -46,7 +46,7 @@ class="img-responsive" alt="Performance over time" /> -**Figure 1**: Average and median normalized query execution times for ClickBench queries for DataFusion 51.0.0 compared to previous releases. +**Figure 1**: Average and median normalized query execution times for ClickBench queries for DataFusion 53.0.0 compared to previous releases. Query times are normalized using the ClickBench definition. See the [DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/) for more details. @@ -65,15 +65,15 @@ matching row groups are skipped entirely.

    Pruning pipeline with limit pruning highlighted -
    Figure 1: Limit pruning is inserted between row group and page index pruning.
    +
    Figure 2: Limit pruning is inserted between row group and page index pruning.
    Thanks to [@xudong963] for implementing this feature. Related PRs: [#18868] ### Improved Filter Pushdown -DataFusion 53 pushes filters down through more join types, and through `UnionExec`, -as well as has expanded support for pushing down [dynamic filters]. More +DataFusion 53 pushes filters down through more join types and through `UnionExec`, +and expands support for pushing down [dynamic filters]. More pushdown means fewer rows flow into joins, repartitions, and later operators, which reduces CPU, memory, and I/O. @@ -99,11 +99,11 @@ driving this work. Related PRs: [#19918], [#20145], [#20192] ### Faster Query Planning -DataFusion 53 improves query planning performance, making immutable pieces of -execution plans cheaper to clone. This helps applications that need extreme low -latency, plan many or complex queries, and those using prepared statements or -parameterized queries. In some benchmarks -overall execution time goes from ~4-5ms to spends ~100us. +DataFusion 53 improves query planning performance by making immutable pieces of +execution plans cheaper to clone. This helps applications that need extremely +low latency, plan many or complex queries, or use prepared statements or +parameterized queries. In some benchmarks, overall execution time drops from +roughly 4-5 ms to about 100 us. Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] @@ -128,7 +128,7 @@ Parquet files. Instead of reading an entire struct column and then extracting the field of interest, DataFusion 53 pushes the field extraction into the scan. For example, the following query reads a struct column `s` and extracts the -`label` field, for rows where the `value` field is greater than 150: +`label` field for rows where the `value` field is greater than 150: ```sql SELECT id, s['label'] @@ -370,5 +370,5 @@ grateful for the time and effort that everyone put in to make it happen. [datafusion-variant]: https://github.com/datafusion-contrib/datafusion-variant [Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ [#21034]: https://github.com/apache/datafusion/issues/21034 -[upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading.html +[upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading/index.html [datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html From c79147d3f53ec3a51e7b7eb6ed27060621ca9954 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 31 Mar 2026 13:27:10 -0400 Subject: [PATCH 16/22] Add known issues --- content/blog/2026-03-25-datafusion-53.0.0.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index efe25f41..cd9b9f8d 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -189,6 +189,13 @@ DataFusion 53 includes some breaking changes, including updates to the SQL parser, optimizer behavior, and some physical-plan APIs. Please see the [upgrade guide] and [changelog] for the full details before upgrading. +## Known Issues + +A small number of issues have been discovered after the 53.0.0 release +and we are planning a 53.1.0 release soon. See the [53.1.0 release tracking +issue] and the related [53.1.0 planning comment] for the latest details. + + ## Thank You Thank you to everyone in the DataFusion community who contributed code, reviews, @@ -370,5 +377,7 @@ grateful for the time and effort that everyone put in to make it happen. [datafusion-variant]: https://github.com/datafusion-contrib/datafusion-variant [Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ [#21034]: https://github.com/apache/datafusion/issues/21034 +[53.1.0 release tracking issue]: https://github.com/apache/datafusion/issues/21079 +[53.1.0 planning comment]: https://github.com/apache/datafusion/issues/21079#issuecomment-4120611441 [upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading/index.html [datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html From e404b057a3eecb0d91a378f6e755ee963ec11091 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 31 Mar 2026 13:30:04 -0400 Subject: [PATCH 17/22] wordsmit --- content/blog/2026-03-25-datafusion-53.0.0.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index cd9b9f8d..962bb68c 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -58,7 +58,7 @@ into earlier and cheaper stages of execution. ### `LIMIT`-Aware Parquet Row Group Pruning DataFusion 53 includes a new optimization that makes Parquet pruning aware of -SQL `LIMIT`, which is described in full in [limit pruning blog post]. If +SQL `LIMIT`. This optimization is described in full in [limit pruning blog post]. If DataFusion can prove that an entire row group matches the predicate, and those fully matching row groups contain enough rows to satisfy the `LIMIT`, partially matching row groups are skipped entirely. @@ -91,7 +91,7 @@ to both sides before the join. This is especially important with [dynamic filter
    Before and after diagram of filter pushdown around a hash join -
    Figure 2: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
    +
    Figure 3: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
    Thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for @@ -109,7 +109,7 @@ Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] ### Faster Functions -DataFusion, like all real-world SQL engines, includes [235 built-in functions], +DataFusion includes [235 built-in functions], and improving the performance of these functions benefits a wide range of workloads. This release includes performance improvements for 42 of those functions, such as [strpos], [replace], [concat], [translate], [array_has], @@ -138,7 +138,7 @@ WHERE s['value'] > 150;
    Before and after diagram of field access pushdown into a data source -
    Figure 3: DataFusion 53 pushes field-access expressions closer to the scan.
    +
    Figure 4: DataFusion 53 pushes field-access expressions closer to the scan.
    Special thanks to [@adriangb] for designing and implementing this optimizer @@ -191,7 +191,7 @@ guide] and [changelog] for the full details before upgrading. ## Known Issues -A small number of issues have been discovered after the 53.0.0 release +A small number of issues were discovered after the 53.0.0 release, and we are planning a 53.1.0 release soon. See the [53.1.0 release tracking issue] and the related [53.1.0 planning comment] for the latest details. From 68e9050b6e94d33b71e9c425eb19421f9332f218 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 31 Mar 2026 13:30:58 -0400 Subject: [PATCH 18/22] wordsmit --- content/blog/2026-03-25-datafusion-53.0.0.md | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 962bb68c..cd3c8de7 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -46,7 +46,7 @@ class="img-responsive" alt="Performance over time" /> -**Figure 1**: Average and median normalized query execution times for ClickBench queries for DataFusion 53.0.0 compared to previous releases. +**Figure 1**: Average and median normalized execution times for DataFusion 53.0.0 on ClickBench queries, compared to previous releases. Query times are normalized using the ClickBench definition. See the [DataFusion Benchmarking Page](https://alamb.github.io/datafusion-benchmarking/) for more details. @@ -58,7 +58,7 @@ into earlier and cheaper stages of execution. ### `LIMIT`-Aware Parquet Row Group Pruning DataFusion 53 includes a new optimization that makes Parquet pruning aware of -SQL `LIMIT`. This optimization is described in full in [limit pruning blog post]. If +`LIMIT`. This optimization is described in full in [limit pruning blog post]. If DataFusion can prove that an entire row group matches the predicate, and those fully matching row groups contain enough rows to satisfy the `LIMIT`, partially matching row groups are skipped entirely. @@ -109,9 +109,8 @@ Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] ### Faster Functions -DataFusion includes [235 built-in functions], -and improving the performance of these functions benefits a wide range of -workloads. This release includes performance improvements for 42 of those +DataFusion includes [235 built-in functions]. Improving the performance of these +functions benefits a wide range of workloads. This release includes performance improvements for 42 of those functions, such as [strpos], [replace], [concat], [translate], [array_has], [array_agg], [left], [right], and [case_when]. @@ -163,7 +162,7 @@ work. Related PRs: [#20065], [#20117], [#20239] * **Spark-Compatible Functions**: This release includes almost 20 new or improved Spark-compatible functions and behaviors in the [datafusion-spark crate]. - This release contains functions such as [collect_list], [date_diff], + It includes functions such as [collect_list], [date_diff], [from_utc_timestamp], [json_tuple], [arrays_zip], [bin], and [array_contains]. Thanks to the contributors who drove this work, especially [@cht42], [@CuteChuanChuan], [@SubhamSinghal], [@kazantsev-maksim], [@unknowntpo], @@ -192,7 +191,7 @@ guide] and [changelog] for the full details before upgrading. ## Known Issues A small number of issues were discovered after the 53.0.0 release, -and we are planning a 53.1.0 release soon. See the [53.1.0 release tracking +and we expect to publish DataFusion 53.1.0 soon. See the [53.1.0 release tracking issue] and the related [53.1.0 planning comment] for the latest details. From 7fac3c4dd37411682945039068fba065fb5aa8a7 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Tue, 31 Mar 2026 13:48:45 -0400 Subject: [PATCH 19/22] Add image styles --- content/blog/2026-03-25-datafusion-53.0.0.md | 23 ++++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index cd3c8de7..32e2a7f6 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -42,7 +42,7 @@ making this release possible. Performance over time @@ -64,7 +64,12 @@ fully matching row groups contain enough rows to satisfy the `LIMIT`, partially matching row groups are skipped entirely.
    -Pruning pipeline with limit pruning highlighted +
    Figure 2: Limit pruning is inserted between row group and page index pruning.
    @@ -90,7 +95,12 @@ Now DataFusion can often transform the physical plan so the filter is applied to both sides before the join. This is especially important with [dynamic filters].
    -Before and after diagram of filter pushdown around a hash join +
    Figure 3: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
    @@ -136,7 +146,12 @@ WHERE s['value'] > 150; ```
    -Before and after diagram of field access pushdown into a data source +
    Figure 4: DataFusion 53 pushes field-access expressions closer to the scan.
    From 5bd1d5cc344855441c0a4aa36f06e9479823b247 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Wed, 1 Apr 2026 12:49:33 -0400 Subject: [PATCH 20/22] Update image --- content/blog/2026-03-25-datafusion-53.0.0.md | 24 ++-- .../join-filter-pushdown.svg | 113 +++++++++++------- 2 files changed, 87 insertions(+), 50 deletions(-) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-03-25-datafusion-53.0.0.md index 32e2a7f6..51b15f37 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-03-25-datafusion-53.0.0.md @@ -86,22 +86,28 @@ For example: ```sql SELECT * -FROM t1 -LEFT JOIN t2 ON t1.k = t2.k -WHERE t1.k = 1; +FROM ( + SELECT * + FROM t1 + LEFT ANTI JOIN t2 ON t1.k = t2.k +) a +JOIN t1 b ON a.k = b.k +WHERE b.v = 1; ``` -Now DataFusion can often transform the physical plan so the filter is applied -to both sides before the join. This is especially important with [dynamic filters]. +Now DataFusion can often transform the physical plan so filters and +[dynamic filters] are pushed deeper into the plan, even through subqueries and +nested joins. In this example, the filter on `b.v` helps derive dynamic filters +that can be pushed into both sides of the nested anti join.
    -
    Figure 3: DataFusion 53 infers additional filters from join conditions and pushes them down in the plan.
    +
    Figure 3: DataFusion 53 pushes dynamic filters through subqueries and into both sides of nested joins.
    Thanks to [@nuno-faria], [@haohuaijin], and [@jackkleeman] for @@ -207,7 +213,7 @@ guide] and [changelog] for the full details before upgrading. A small number of issues were discovered after the 53.0.0 release, and we expect to publish DataFusion 53.1.0 soon. See the [53.1.0 release tracking -issue] and the related [53.1.0 planning comment] for the latest details. +issue] for more details. ## Thank You diff --git a/content/images/datafusion-53.0.0/join-filter-pushdown.svg b/content/images/datafusion-53.0.0/join-filter-pushdown.svg index 68bd1101..141dd827 100644 --- a/content/images/datafusion-53.0.0/join-filter-pushdown.svg +++ b/content/images/datafusion-53.0.0/join-filter-pushdown.svg @@ -1,4 +1,4 @@ - + @@ -6,63 +6,94 @@ - - Before - + Before + Dynamic filters stop at the subquery boundary + - - FilterExec - k = 1 + + HashJoinExec + a.k = b.k - + + - - HashJoinExec + + Subquery a - - + + HashJoinExec + LeftAnti - - DataSourceExec - t1 + + - - DataSourceExec - t2 + + DataSourceExec + t1 - - After - + + DataSourceExec + t2 - - HashJoinExec + + FilterExec + b.v = 1 - - + - - FilterExec - k = 1 + + DataSourceExec + t1 as b - - FilterExec - k = 1 + No dynamic filters reach the nested join inputs - - + After + Dynamic filters pushed through the subquery into both scans + - - DataSourceExec - t1 + + HashJoinExec + a.k = b.k - - DataSourceExec - t2 + + + + + Subquery a + + + HashJoinExec + LeftAnti + + + + + + DataSourceExec + t1 IN <Dynamic Filter> + + + DataSourceExec + t2 IN <Dynamic Filter> + + + FilterExec + b.v = 1 + + + + + DataSourceExec + t1 as b + + Dynamic filters are pushed into both sides of the nested anti join From 01c177ad2e6ee940943723a611fe4a67d55cb367 Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 2 Apr 2026 08:44:27 -0400 Subject: [PATCH 21/22] Update Date to 2026-04-02 --- ...-25-datafusion-53.0.0.md => 2026-04-02-datafusion-53.0.0.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename content/blog/{2026-03-25-datafusion-53.0.0.md => 2026-04-02-datafusion-53.0.0.md} (99%) diff --git a/content/blog/2026-03-25-datafusion-53.0.0.md b/content/blog/2026-04-02-datafusion-53.0.0.md similarity index 99% rename from content/blog/2026-03-25-datafusion-53.0.0.md rename to content/blog/2026-04-02-datafusion-53.0.0.md index 51b15f37..ed3e41a5 100644 --- a/content/blog/2026-03-25-datafusion-53.0.0.md +++ b/content/blog/2026-04-02-datafusion-53.0.0.md @@ -1,7 +1,7 @@ --- layout: post title: Apache DataFusion 53.0.0 Released -date: 2026-03-25 +date: 2026-04-02 author: pmc categories: [release] --- From b4a58fcae4d6c86cab4b342ddcd6cd065031b66c Mon Sep 17 00:00:00 2001 From: Andrew Lamb Date: Thu, 2 Apr 2026 08:48:11 -0400 Subject: [PATCH 22/22] last minute obsession cleanups --- content/blog/2026-04-02-datafusion-53.0.0.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/content/blog/2026-04-02-datafusion-53.0.0.md b/content/blog/2026-04-02-datafusion-53.0.0.md index ed3e41a5..030e4f37 100644 --- a/content/blog/2026-04-02-datafusion-53.0.0.md +++ b/content/blog/2026-04-02-datafusion-53.0.0.md @@ -61,7 +61,7 @@ DataFusion 53 includes a new optimization that makes Parquet pruning aware of `LIMIT`. This optimization is described in full in [limit pruning blog post]. If DataFusion can prove that an entire row group matches the predicate, and those fully matching row groups contain enough rows to satisfy the `LIMIT`, partially -matching row groups are skipped entirely. +matching row groups are skipped entirely.
    @@ -126,7 +126,7 @@ Thanks to [@askalt] for leading this work. Related PRs: [#19792], [#19893] ### Faster Functions DataFusion includes [235 built-in functions]. Improving the performance of these -functions benefits a wide range of workloads. This release includes performance improvements for 42 of those +functions benefits a wide range of workloads. This release improves the performance of 42 of those functions, such as [strpos], [replace], [concat], [translate], [array_has], [array_agg], [left], [right], and [case_when]. @@ -213,7 +213,7 @@ guide] and [changelog] for the full details before upgrading. A small number of issues were discovered after the 53.0.0 release, and we expect to publish DataFusion 53.1.0 soon. See the [53.1.0 release tracking -issue] for more details. +issue] for the latest status. ## Thank You @@ -398,6 +398,5 @@ grateful for the time and effort that everyone put in to make it happen. [Parquet Variant]: https://parquet.apache.org/blog/2026/02/27/variant-type-in-apache-parquet-for-semi-structured-data/ [#21034]: https://github.com/apache/datafusion/issues/21034 [53.1.0 release tracking issue]: https://github.com/apache/datafusion/issues/21079 -[53.1.0 planning comment]: https://github.com/apache/datafusion/issues/21079#issuecomment-4120611441 [upgrade guide]: https://datafusion.apache.org/library-user-guide/upgrading/index.html [datafusion-spark crate]: https://docs.rs/datafusion-spark/latest/datafusion_spark/index.html