You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/blog/2025-03-11-ordering-analysis.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,7 @@ limitations under the License.
31
31
32
32
## Introduction
33
33
In this blog post, we explain when an ordering requirement of an operator is satisfied by its input data. This analysis is essential for order-based optimizations and is often more complex than one might initially think.
<strong>Ordering Requirement</strong> for an operator describes how the input data to that operator must be sorted for the operator to compute the correct result. It is the job of the planner to make sure that these requirements are satisfied during execution (See DataFusion <a href="https://docs.rs/datafusion/latest/datafusion/physical_optimizer/enforce_sorting/struct.EnforceSorting.html" target="_blank">EnforceSorting</a> for an implementation of such a rule).
36
36
</blockquote>
37
37
@@ -134,7 +134,7 @@ Let's start by creating an example table that we will refer throughout the post.
<strong>How can a table have multiple orderings?</strong> At first glance it may seem counterintuitive for a table to have more than one valid ordering. However, during query execution such scenarios can arise.
139
139
140
140
For example consider the following query:
@@ -197,7 +197,7 @@ To solve the shortcomings above DataFusion needs to track of following propertie
197
197
- Equivalent Expression Groups (will be explained shortly)
198
198
- Succinct Valid Orderings (will be explained shortly)
<strong>Note:</strong> These properties are implemented in the <code>EquivalenceProperties</code> structure in <code>DataFusion</code>, please see the <a href="https://github.com/apache/datafusion/blob/f47ea73b87eec4af044f9b9923baf042682615b2/datafusion/physical-expr/src/equivalence/properties/mod.rs#L134" target="_blank">source</a> for more details<br>
202
202
</blockquote>
203
203
@@ -210,7 +210,7 @@ For instance in the example table:
210
210
211
211
- Columns `hostname` and `currency` are constant because every row in the table has the same value (`'app.example.com'` for `hostname`, and `'USD'` for `currency`) for these columns.
<strong>Note:</strong> Constant expressions can arise during query execution. For example, in following query:<br>
215
215
<code>SELECT hostname FROM logs</code><br><code>WHERE hostname='app.example.com'</code> <br>
216
216
after filtering is done, for subsequent operators the <code>hostname</code> column will be constant.
@@ -221,7 +221,7 @@ Equivalent expression groups are expressions that always hold the same value acr
221
221
222
222
In the example table, the expressions `price` and `price_cloned` form one equivalence group, and `time` and `time_cloned` form another equivalence group.
<strong>Note:</strong> Equivalent expression groups can arise during the query execution. For example, in the following query:<br>
226
226
<code>SELECT time, time as time_cloned FROM logs</code> <br>
227
227
after the projection is done, for subsequent operators <code>time</code> and <code>time_cloned</code> will form an equivalence group. As another example, in the following query:<br>
@@ -293,7 +293,7 @@ Following third and fourth constraints for the simplified table, the succinct va
<p><strong>How can DataFusion find orderings?</strong></p>
298
298
DataFusion's <code>CREATE EXTERNAL TABLE</code> has a <code>WITH ORDER</code> clause (see <ahref="https://datafusion.apache.org/user-guide/sql/ddl.html#create-external-table">docs</a>) to specify the known orderings of the table during table creation. For example the following query:<br>
Copy file name to clipboardExpand all lines: content/blog/2025-06-30-cancellation.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -359,7 +359,7 @@ To illustrate what this process looks like, let's have a look at the execution o
359
359
If we assume a task budget of 1 unit, each time Tokio schedules the task would result in the following sequence of function calls.
360
360
361
361
<figure>
362
-
<img src="/blog/images/task-cancellation/tokio_budget.png" style="width: 100%; max-width: 100%" class="img-fluid" alt="Sequence diagram showing how the tokio task budget is used and reset."
362
+
<img src="/blog/images/task-cancellation/tokio_budget.png" class="img-fluid" alt="Sequence diagram showing how the tokio task budget is used and reset."
363
363
/>
364
364
<figcaption>Tokio task budget system, assuming the task budget is set to 1, for the plan above.</figcaption>
Databases are some of the most complex yet interesting pieces of software. They are amazing pieces of abstraction: query engines optimize and execute complex plans, storage engines provide sophisticated infrastructure as the backbone of the system, while intricate file formats lay the groundwork for particular workloads. All of this is exposed by a user-friendly interface and query languages (typically a dialect of SQL).
32
32
<br><br>
33
33
Starting a journey learning about database internals can be daunting. With so many topics that are whole PhD degrees themselves, finding a place to start is difficult. In this blog post, I will share my early journey in the database world and a quick lesson on one of the first topics I dove into. If you are new to the space, this post will help you get your first foot into the database world, and if you are already a veteran, you may still learn something new.
Round-robin repartitioning is the simplest partitioning strategy. Incoming data is processed in batches (chunks of rows), and these batches are distributed across partitions cyclically or sequentially, with each new batch assigned to the next available partition.
129
128
<br><br>
130
129
Round-robin repartitioning is useful when the data grouping isn't known or when aiming for an even distribution across partitions. Because it simply assigns batches in order without inspecting their contents, it is a low-overhead way to increase parallelism for downstream operations.
Hash repartitioning distributes data based on a hash function applied to one or more columns, called the partitioning key. Rows with the same hash value are placed in the same partition.
149
147
<br><br>
150
148
Hash repartitioning is useful when working with grouped data. Imagine you have a database containing information on company sales, and you are looking to find the total revenue each store produced. Hash repartitioning would make this query much more efficient. Rather than iterating over the data on a single thread and keeping a running sum for each store, it would be better to hash repartition on the store column and have multiple threads calculate individual store sales.
0 commit comments