Skip to content

[SPARK-56121][SQL] Use dedicated EmptyRelation node for static PropagateEmptyRelation#55014

Open
liuzqt wants to merge 3 commits intoapache:masterfrom
liuzqt:SPARK-56121
Open

[SPARK-56121][SQL] Use dedicated EmptyRelation node for static PropagateEmptyRelation#55014
liuzqt wants to merge 3 commits intoapache:masterfrom
liuzqt:SPARK-56121

Conversation

@liuzqt
Copy link
Contributor

@liuzqt liuzqt commented Mar 25, 2026

What changes were proposed in this pull request?

In previous PR #46830, I introduce dedicated node for empty relation

  • EmptyRelation logical node
  • EmptyRelationExec physical node
    and wrap the eliminated plan segment in AQEPropagateEmptyRelation

In this PR, I extend this change to static PropagateEmptyRelation rule.

Note: this doesn't 100% remove the usage of empty LocalRelation for empty, there're still some other rules which can produce empty LocalRelation, and we still recognize the empty LocalRelation as an empty relation.

Why are the changes needed?

Better observability, in both spark UI, and explain plan string, we're able to see the eliminated plan segment.

Plan string

  • before
== Physical Plan ==
LocalTableScan (1)


(1) LocalTableScan
Output [2]: [k#1L, v#7]
Arguments: <empty>, [k#1L, v#7]

  • after
== Physical Plan ==
EmptyRelation (1)
+- Project [k#1L, 1 AS v#7]
   +- EmptyRelation
      +- Join Inner, (k#1L = k#6L)
         :- RepartitionByExpression [k#1L], 8
         :  +- Project [id#0L AS k#1L]
         :     +- Range (0, 50000, step=1, splits=Some(32))
         +- LocalRelation <empty>, [k#6L]


(1) EmptyRelation
Output [2]: [k#1L, v#7]
Arguments: [plan_id=2]

Spark UI

  • before
Screenshot 2026-03-23 at 7 45 26 PM
  • after
Screenshot 2026-03-23 at 8 21 05 PM

Does this PR introduce any user-facing change?

Yes

  • spark ui change
  • explain string change

How was this patch tested?

  • updated exising UTs
  • verified Spark UI change

Was this patch authored or co-authored using generative AI tooling?

Yes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant