Skip to content

Include null_aware status in the relevant Join node display implementations#22913

Open
AdamGS wants to merge 3 commits into
apache:mainfrom
AdamGS:adamg/null-aware-hash-join-display
Open

Include null_aware status in the relevant Join node display implementations#22913
AdamGS wants to merge 3 commits into
apache:mainfrom
AdamGS:adamg/null-aware-hash-join-display

Conversation

@AdamGS

@AdamGS AdamGS commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

This change makes testing null_aware behavior easier, and also makes the performance of various joins clearer - null_aware joins do extra work.

This was originally part #21585, but it seems like there is a bunch of activity around null-aware joins, so I figured its worth splitting out.

What changes are included in this PR?

Add a null_aware indication to relevant Display implementations when appropriate.

Are these changes tested?

SLT tests

Are there any user-facing changes?

Display only

@github-actions github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) physical-plan Changes to the physical-plan crate labels Jun 11, 2026
AdamGS added 2 commits June 11, 2026 18:34
Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
@AdamGS AdamGS force-pushed the adamg/null-aware-hash-join-display branch from 0ed8e88 to ed0b9d1 Compare June 11, 2026 17:58

@2010YOUY01 2010YOUY01 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you! Left one minor suggestion for extra test.

writeln!(f, "NullsEqual: true")?;
}

if self.null_aware {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add test for this

-- in slt
explain format tree
select...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, put it in explain_tree.slt.

Signed-off-by: Adam Gutglick <adamgsal@gmail.com>
@github-actions

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning apache/main
    Building datafusion-expr v54.0.0 (current)
       Built [  27.009s] (current)
     Parsing datafusion-expr v54.0.0 (current)
      Parsed [   0.073s] (current)
    Building datafusion-expr v54.0.0 (baseline)
       Built [  26.806s] (baseline)
     Parsing datafusion-expr v54.0.0 (baseline)
      Parsed [   0.075s] (baseline)
    Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.420s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  57.236s] datafusion-expr
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  37.862s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.128s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  37.335s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.130s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.609s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  77.733s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
error: running cargo-doc on crate 'datafusion-sqllogictest' failed with output:
-----
   Compiling proc-macro2 v1.0.106
   Compiling quote v1.0.45
   Compiling unicode-ident v1.0.24
   Compiling libc v0.2.186
    Checking cfg-if v1.0.4
    Checking bytes v1.11.1
    Checking memchr v2.8.2
   Compiling serde_core v1.0.228
   Compiling syn v2.0.117
   Compiling autocfg v1.5.1
   Compiling shlex v2.0.1
   Compiling jobserver v0.1.34
   Compiling find-msvc-tools v0.1.9
   Compiling cc v1.2.64
    Checking itoa v1.0.18
    Checking equivalent v1.0.2
    Checking foldhash v0.2.0
    Checking allocator-api2 v0.2.21
    Checking once_cell v1.21.4
   Compiling libm v0.2.16
   Compiling num-traits v0.2.19
    Checking hashbrown v0.17.1
    Checking pin-project-lite v0.2.17
   Compiling zmij v1.0.21
    Checking indexmap v2.14.0
   Compiling zerocopy v0.8.52
    Checking futures-core v0.3.32
    Checking futures-sink v0.3.32
    Checking errno v0.3.14
    Checking signal-hook-registry v1.4.8
    Checking mio v1.2.1
    Checking socket2 v0.6.4
   Compiling version_check v0.9.5
   Compiling serde v1.0.228
   Compiling getrandom v0.3.4
    Checking num-integer v0.1.46
   Compiling serde_json v1.0.150
    Checking slab v0.4.12
    Checking futures-channel v0.3.32
    Checking smallvec v1.15.2
    Checking num-bigint v0.4.6
   Compiling synstructure v0.13.2
    Checking http v1.4.2
    Checking base64 v0.22.1
    Checking futures-io v0.3.32
    Checking futures-task v0.3.32
    Checking iana-time-zone v0.1.65
    Checking tracing-core v0.1.36
    Checking chrono v0.4.45
   Compiling zerocopy-derive v0.8.52
   Compiling tokio-macros v2.7.0
    Checking tokio v1.52.3
   Compiling serde_derive v1.0.228
   Compiling futures-macro v0.3.32
    Checking futures-util v0.3.32
   Compiling tracing-attributes v0.1.31
   Compiling zerofrom-derive v0.1.7
    Checking siphasher v1.0.3
    Checking tracing v0.1.44
   Compiling yoke-derive v0.8.2
    Checking stable_deref_trait v1.2.1
    Checking num-complex v0.4.6
   Compiling zerovec-derive v0.11.3
    Checking zerofrom v0.1.8
    Checking getrandom v0.2.17
   Compiling pkg-config v0.3.33
    Checking phf_shared v0.12.1
    Checking half v2.7.1
   Compiling displaydoc v0.2.6
   Compiling ahash v0.8.12
   Compiling chrono-tz v0.10.4
    Checking yoke v0.8.3
    Checking percent-encoding v2.3.2
   Compiling thiserror v2.0.18
    Checking zerovec v0.11.6
    Checking arrow-buffer v59.0.0
    Checking phf v0.12.1
    Checking ppv-lite86 v0.2.21
   Compiling thiserror-impl v2.0.18
    Checking arrow-schema v59.0.0
   Compiling ring v0.17.14
    Checking arrow-data v59.0.0
   Compiling semver v1.0.28
    Checking rand_core v0.10.1
   Compiling getrandom v0.4.2
    Checking log v0.4.32
    Checking tinystr v0.8.3
    Checking litemap v0.8.2
    Checking untrusted v0.9.0
    Checking writeable v0.6.3
    Checking bitflags v2.13.0
    Checking icu_locale_core v2.2.0
    Checking potential_utf v0.1.5
    Checking zerotrie v0.2.4
   Compiling zstd-sys v2.0.16+zstd.1.5.7
   Compiling async-trait v0.1.89
    Checking rand_core v0.9.5
   Compiling icu_properties_data v2.2.0
    Checking utf8_iter v1.0.4
   Compiling icu_normalizer_data v2.2.0
    Checking icu_collections v2.2.0
    Checking rand_chacha v0.9.0
    Checking icu_provider v2.2.0
    Checking rand v0.9.4
    Checking aho-corasick v1.1.4
    Checking ryu v1.0.23
   Compiling object v0.37.3
   Compiling zstd-safe v7.2.4
    Checking regex-syntax v0.8.11
    Checking arrow-array v59.0.0
    Checking lexical-util v1.0.7
    Checking regex-automata v0.4.14
    Checking arrow-select v59.0.0
    Checking icu_normalizer v2.2.0
    Checking icu_properties v2.2.0
   Compiling rustix v1.1.4
    Checking regex v1.12.4
    Checking form_urlencoded v1.2.2
    Checking unicode-width v0.2.2
    Checking idna_adapter v1.2.2
   Compiling crc32fast v1.5.0
   Compiling parking_lot_core v0.9.12
    Checking either v1.16.0
    Checking typenum v1.20.1
    Checking idna v1.1.0
    Checking lexical-write-integer v1.0.6
    Checking lexical-parse-integer v1.0.6
   Compiling rustc_version v0.4.1
    Checking futures-executor v0.3.32
   Compiling pin-project-internal v1.1.13
    Checking scopeguard v1.2.0
    Checking adler2 v2.0.1
    Checking simd-adler32 v0.3.9
    Checking miniz_oxide v0.8.9
    Checking lock_api v0.4.14
    Checking futures v0.3.32
   Compiling flatbuffers v25.12.19
    Checking lexical-parse-float v1.0.6
    Checking pin-project v1.1.13
    Checking lexical-write-float v1.0.6
    Checking url v2.5.8
    Checking zlib-rs v0.6.3
    Checking unicode-segmentation v1.13.3
   Compiling ar_archive_writer v0.5.2
    Checking comfy-table v7.2.2
    Checking lexical-core v1.0.6
    Checking itertools v0.14.0
    Checking flate2 v1.1.9
   Compiling psm v0.1.31
    Checking arrow-ord v59.0.0
    Checking twox-hash v2.1.2
    Checking atoi v2.0.0
   Compiling stacker v0.1.24
   Compiling snap v1.1.1
    Checking hex v0.4.3
    Checking alloc-no-stdlib v2.0.4
    Checking lz4_flex v0.13.1
    Checking alloc-stdlib v0.2.2
    Checking arrow-cast v59.0.0
    Checking parking_lot v0.12.5
    Checking csv-core v0.1.13
   Compiling paste v1.0.15
    Checking same-file v1.0.6
    Checking simdutf8 v0.1.5
    Checking humantime v2.3.0
    Checking walkdir v2.5.0
    Checking csv v1.4.0
    Checking brotli-decompressor v5.0.1
   Compiling recursive-proc-macro-impl v0.1.1
    Checking subtle v2.6.1
    Checking arrow-json v59.0.0
    Checking recursive v0.1.1
    Checking brotli v8.0.3
    Checking arrow-csv v59.0.0
    Checking object_store v0.13.2
    Checking arrow-string v59.0.0
    Checking arrow-row v59.0.0
    Checking arrow-arith v59.0.0
    Checking uuid v1.23.3
    Checking tokio-util v0.7.18
    Checking zstd v0.13.3
   Compiling sqlparser_derive v0.5.0
    Checking arrow-ipc v59.0.0
   Compiling seq-macro v0.3.6
    Checking cpufeatures v0.3.0
    Checking hybrid-array v0.4.12
    Checking linux-raw-sys v0.12.1
    Checking sqlparser v0.62.0
    Checking cmov v0.5.4
    Checking ctutils v0.4.2
    Checking crypto-common v0.2.2
    Checking block-buffer v0.12.1
    Checking arrow v59.0.0
    Checking parquet v59.0.0
   Compiling generic-array v0.14.7
    Checking const-oid v0.10.2
    Checking digest v0.11.3
    Checking datafusion-doc v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/doc)
   Compiling crossbeam-utils v0.8.21
    Checking foldhash v0.1.5
    Checking block-buffer v0.10.4
    Checking hashbrown v0.15.5
    Checking crypto-common v0.1.7
    Checking fixedbitset v0.5.7
    Checking fastrand v2.4.1
   Compiling heck v0.5.0
    Checking petgraph v0.8.3
    Checking tempfile v3.27.0
    Checking digest v0.10.7
    Checking sha2 v0.11.0
    Checking md-5 v0.11.0
    Checking hashbrown v0.14.5
    Checking dashmap v6.2.1
   Compiling datafusion-macros v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/macros)
   Compiling blake3 v1.8.5
   Compiling anyhow v1.0.102
    Checking arrayref v0.3.9
    Checking constant_time_eq v0.4.2
    Checking arrayvec v0.7.6
    Checking blake2 v0.10.6
   Compiling liblzma-sys v0.4.6
    Checking libbz2-rs-sys v0.2.5
    Checking bzip2 v0.6.1
    Checking datafusion-common-runtime v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common-runtime)
    Checking compression-core v0.4.32
   Compiling httparse v1.10.1
   Compiling prost-derive v0.14.4
    Checking http-body v1.0.1
    Checking glob v0.3.3
    Checking tower-service v0.3.3
    Checking atomic-waker v1.1.2
    Checking fnv v1.0.7
    Checking try-lock v0.2.5
    Checking want v0.3.1
    Checking h2 v0.4.14
    Checking tokio-stream v0.1.18
    Checking httpdate v1.0.3
    Checking zeroize v1.8.2
    Checking rustls-pki-types v1.14.1
    Checking hyper v1.10.1
   Compiling prost v0.14.4
    Checking datafusion-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/common)
    Checking hyper-util v0.1.20
    Checking http-body-util v0.1.3
    Checking sync_wrapper v1.0.2
   Compiling prettyplease v0.2.37
   Compiling rustls v0.23.40
    Checking tower-layer v0.3.3
   Compiling prost-types v0.14.4
    Checking liblzma v0.4.6
    Checking compression-codecs v0.4.38
    Checking async-compression v0.4.42
    Checking rustls-webpki v0.103.13
   Compiling serde_derive_internals v0.29.1
    Checking datafusion-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr-common)
    Checking mime v0.3.17
   Compiling schemars v0.8.22
   Compiling schemars_derive v0.8.22
   Compiling hashbrown v0.16.1
    Checking datafusion-physical-expr-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-common)
    Checking axum-core v0.5.6
    Checking datafusion-functions-window-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window-common)
    Checking datafusion-functions-aggregate-common v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate-common)
    Checking datafusion-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/expr)
    Checking tower v0.5.3
   Compiling multimap v0.10.1
    Checking matchit v0.8.4
   Compiling dyn-clone v1.0.20
    Checking axum v0.8.9
   Compiling prost-build v0.14.4
   Compiling regress v0.10.5
   Compiling pbjson-build v0.8.0
    Checking datafusion-physical-expr v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr)
    Checking datafusion-execution v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/execution)
    Checking hyper-timeout v0.5.2
    Checking datafusion-functions v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions)
    Checking chacha20 v0.10.0
   Compiling strsim v0.11.1
   Compiling portable-atomic v1.13.1
   Compiling ident_case v1.0.1
   Compiling darling_core v0.23.0
   Compiling typify-impl v0.5.0
    Checking rand v0.10.1
    Checking tonic v0.14.6
   Compiling serde_tokenstream v0.2.3
    Checking ureq-proto v0.6.0
   Compiling bigdecimal v0.4.10
    Checking utf8parse v0.2.2
    Checking crc-catalog v2.5.0
    Checking tinyvec_macros v0.1.1
    Checking utf8-zero v0.8.1
   Compiling bollard-buildkit-proto v0.7.0
    Checking datafusion-physical-plan v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-plan)
    Checking datafusion-physical-expr-adapter v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-expr-adapter)
    Checking ureq v3.3.0
    Checking tinyvec v1.11.0
    Checking crc v3.4.0
    Checking anstyle-parse v1.0.0
   Compiling darling_macro v0.23.0
    Checking tonic-prost v0.14.6
    Checking datafusion-functions-aggregate v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-aggregate)
   Compiling strum_macros v0.28.0
   Compiling typify-macro v0.5.0
   Compiling structmeta-derive v0.3.0
    Checking deranged v0.5.8
    Checking num-conv v0.2.2
    Checking powerfmt v0.2.0
    Checking is_terminal_polyfill v1.70.2
   Compiling unsafe-libyaml v0.2.11
    Checking time-core v0.1.9
    Checking colorchoice v1.0.5
    Checking anstyle-query v1.1.5
    Checking anstyle v1.0.14
    Checking anstream v1.0.0
    Checking time v0.3.48
   Compiling serde_yaml v0.9.34+deprecated
   Compiling structmeta v0.3.0
    Checking arrow-avro v59.0.0
    Checking datafusion-functions-nested v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-nested)
   Compiling typify v0.5.0
   Compiling darling v0.23.0
    Checking datafusion-session v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/session)
    Checking datafusion-datasource v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource)
    Checking unicode-normalization v0.1.25
   Compiling pbjson-types v0.8.0
    Checking tokio-rustls v0.26.4
   Compiling serde_repr v0.1.20
   Compiling async-stream-impl v0.3.6
    Checking num-rational v0.4.2
    Checking datafusion-catalog v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/catalog)
    Checking datafusion-pruning v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/pruning)
    Checking num-iter v0.1.45
    Checking unicode-bidi v0.3.18
    Checking unicode-properties v0.1.4
    Checking openssl-probe v0.2.1
    Checking clap_lex v1.1.0
    Checking datafusion-datasource-parquet v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-parquet)
    Checking datafusion-physical-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/physical-optimizer)
    Checking clap_builder v4.6.0
    Checking stringprep v0.1.5
    Checking rustls-native-certs v0.8.4
    Checking num v0.4.3
    Checking bollard-stubs v1.52.1-rc.29.1.3
    Checking async-stream v0.3.6
    Checking datafusion-datasource-arrow v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-arrow)
    Checking datafusion-datasource-json v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-json)
    Checking datafusion-datasource-avro v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-avro)
    Checking datafusion-datasource-csv v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/datasource-csv)
    Checking datafusion-sql v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/sql)
    Checking hyper-rustls v0.27.9
   Compiling serde_with_macros v3.21.0
   Compiling substrait v0.63.0
   Compiling parse-display-derive v0.9.1
    Checking datafusion-functions-window v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/functions-window)
    Checking datafusion-optimizer v54.0.0 (/home/runner/work/datafusion/datafusion/datafusion/optimizer)
    Checking hyperlocal v0.9.1
   Compiling clap_derive v4.6.1
    Checking xattr v1.6.1
    Checking hmac v0.13.0
    Checking serde_urlencoded v0.7.1
    Checking pbjson v0.8.0
    Checking rand_core v0.6.4
   Compiling enum-ordinalize-derive v4.3.2
   Compiling fs-err v3.3.0
    Checking filetime v0.2.29
    Checking home v0.5.12
    Checking rustc-hash v2.1.2
    Checking byteorder v1.5.0
   Compiling owo-colors v4.3.0
    Checking fallible-iterator v0.2.0
    Checking postgres-protocol v0.6.12
    Checking astral-tokio-tar v0.6.2
    Checking clap v4.6.1
   Compiling enum-ordinalize v4.3.2
    Checking rand_chacha v0.3.1
    Checking parse-display v0.9.1
    Checking bollard v0.20.2
error[E0119]: conflicting implementations of trait `From<format_description::parse::format_item::HourBase>` for type `<format_description::parse::format_item::HourBase as format_description::parse::format_item::ModifierValue>::Type`
   --> /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/bollard-0.20.2/src/docker.rs:209:1
    |
209 | impl<T: Into<String>> From<T> for MaybeClientVersion {
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: conflicting implementation in crate `time`:
            - impl From<format_description::parse::format_item::HourBase> for <format_description::parse::format_item::HourBase as format_description::parse::format_item::ModifierValue>::Type;
    = note: upstream crates may add a new impl of trait `std::convert::Into<std::string::String>` for type `time::format_description::parse::format_item::HourBase` in future versions

    Checking serde_with v3.21.0
    Checking ferroid v2.0.0
   Compiling postgres-derive v0.4.9
For more information about this error, try `rustc --explain E0119`.
error: could not compile `bollard` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

-----

error: failed to build rustdoc for crate datafusion-sqllogictest v54.0.0
note: this is usually due to a compilation error in the crate,
      and is unlikely to be a bug in cargo-semver-checks
note: the following command can be used to reproduce the error:
      cargo new --lib example &&
          cd example &&
          echo '[workspace]' >> Cargo.toml &&
          cargo add --path /home/runner/work/datafusion/datafusion/datafusion/sqllogictest --features avro,backtrace,bytes,chrono,datafusion-substrait,parquet_encryption,postgres,postgres-types,substrait,testcontainers-modules,tokio-postgres &&
          cargo check &&
          cargo doc

error: aborting due to failure to build rustdoc for crate datafusion-sqllogictest v54.0.0

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Jun 12, 2026
@AdamGS

AdamGS commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

semver check failure is a false positive, see #22933

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change logical-expr Logical plan and expressions physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Null-aware joins don't display that property

2 participants