Fix nullable IN-list runtime semantics

[LeftHandCold]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue #25346

Follow-up to #25427.

What this PR does / why we need it:

The previous zonemap fix made block pruning conservative for nullable IN lists, but the SQL path can still fail before row-level filtering. Planner constant folding converts IN list literals into a folded vector. Vector.InplaceSortAndCompact() sorts the vector payload without moving the null bitmap, so ('keep', NULL) can become a vector where the null bitmap no longer points at the same value. The runtime in operator then skips the wrong RHS slot while building its hash map, and WHERE k IN ('keep', NULL) can filter out matching keep rows.

This PR fixes the full path:

• keeps folded nullable IN-list vectors unsorted/uncompacted so values and null bitmap stay aligned;
• makes vectorized IN / NOT IN return SQL UNKNOWN for no-match rows when the RHS contains NULL;
• keeps nullable folded vectors conservative in zonemap/readutil pruning instead of relying on sorted first/last entries;
• adds tests for operator three-valued logic, folded predicate execution, nullable folded vectors in zonemap pruning, and varchar/fixed-type nullable zonemap AnyIn.

Validated with the issue-shaped SQL on a dev node:

SELECT COUNT(*) FROM t WHERE k IN ('keep', NULL);
-- 2

SELECT COUNT(*) FROM t WHERE k NOT IN ('zzz', NULL);
-- 0

Local checks:

go test ./pkg/sql/plan/function -count=1
go test ./pkg/sql/colexec -count=1
go test ./pkg/vm/engine/readutil -count=1
go test ./pkg/vm/engine/tae/index -count=1
go test ./pkg/sql/plan/rule -count=1
go test ./pkg/sql/plan -run 'Test.*Fold|Test.*In|Test.*Filter' -count=1

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[XuPeng-SH]

Reviewed the nullable IN-list runtime and pruning fix.

The direction looks correct. The runtime IN/NOT IN operators now preserve SQL three-valued logic when the RHS contains NULL: matches still return true/false as expected, while no-match rows become UNKNOWN. The folded-vector change also addresses the real root cause: nullable list vectors are no longer sorted/compacted in a way that can desynchronize payload order from the null bitmap.

I also checked the pruning paths. For nullable folded vectors, readutil disables sorted min/max seek shortcuts, and ZM.AnyIn falls back to scanning non-null values. SubVecIn returning the full range for nullable vectors is conservative for bloom filtering; it can add false positives but should not incorrectly prune matching rows.

Local validation:

• go test ./pkg/sql/plan/function -count=1
• go test ./pkg/sql/colexec -count=1
• go test ./pkg/vm/engine/readutil -count=1
• go test ./pkg/vm/engine/tae/index -count=1
• go test ./pkg/sql/plan/rule -count=1
• go test ./pkg/sql/plan -run 'Test.*Fold|Test.*In|Test.*Filter' -count=1

CI is green. I did not find a blocking issue.

[mergify]

Merge Queue Status

• ✅ Entered queue — 2026-07-06 11:23 UTC · Rule: main · triggered by rule Automatic queue on approval for main
• ✅ Checks passed · in-place
• ✅ Merged — 2026-07-06 12:59 UTC · at 3b347d35f089764a28c74dd30b96465f441e5e4a · squash

This pull request spent 1 hour 35 minutes 22 seconds in the queue, including 1 hour 35 minutes 7 seconds running CI.

Required conditions to merge

• #review-threads-unresolved = 0 [🛡 GitHub branch protection]
  • Fix nullable IN-list runtime semantics #25467
• github-review-approved [🛡 GitHub branch protection]
  • Fix nullable IN-list runtime semantics #25467
• github-review-decision = APPROVED [🛡 GitHub branch protection]
  • Fix nullable IN-list runtime semantics #25467
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-neutral = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
  • check-skipped = Matrixone Standlone CI / Multi-CN e2e BVT Test on Linux/x64(LAUNCH, PROXY)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH, PESSIMISTIC)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / UT Test on Ubuntu/x86
  • check-neutral = Matrixone CI / UT Test on Ubuntu/x86
  • check-skipped = Matrixone CI / UT Test on Ubuntu/x86
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
  • check-neutral = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
  • check-skipped = Matrixone Compose CI / multi cn e2e bvt test docker compose(Optimistic/PUSH)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
  • check-neutral = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
  • check-skipped = Matrixone Standlone CI / e2e BVT Test on Linux/x64(LAUNCH,Optimistic)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
  • check-neutral = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
  • check-skipped = Matrixone Upgrade CI / Compatibility Test With Target on Linux/x64(LAUNCH)
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone Utils CI / Coverage
  • check-neutral = Matrixone Utils CI / Coverage
  • check-skipped = Matrixone Utils CI / Coverage
• any of [🛡 GitHub branch protection]:
  • check-success = Matrixone CI / SCA Test on Linux/arm64
  • check-neutral = Matrixone CI / SCA Test on Linux/arm64
  • check-skipped = Matrixone CI / SCA Test on Linux/arm64