Skip to content

[SPARK-57576][SQL] Generalize error messages for set operations on MAP and VARIANT types#56614

Closed
KenanAdel wants to merge 2 commits into
apache:masterfrom
KenanAdel:SPARK-57576
Closed

[SPARK-57576][SQL] Generalize error messages for set operations on MAP and VARIANT types#56614
KenanAdel wants to merge 2 commits into
apache:masterfrom
KenanAdel:SPARK-57576

Conversation

@KenanAdel

Copy link
Copy Markdown

What changes were proposed in this pull request?

This PR updates the error messages for SET_OPERATION_ON_MAP_TYPE and SET_OPERATION_ON_VARIANT_TYPE in error-conditions.json.

Previously, the messages incorrectly implied that the limitation only applies to DataFrame operations. In reality, the same error is also raised when using SQL queries.

This change rewrites the messages to be more general so they correctly describe the restriction for both DataFrame and SQL when performing set operations (like INTERSECT and EXCEPT) on MAP and VARIANT types.


Why are the changes needed?

The current wording is misleading because it focuses only on DataFrame usage.

However, these errors also happen in SQL execution, so the message should reflect both cases. This update removes confusion and makes the behavior clearer and more accurate for users.


Does this PR introduce any user-facing change?

Yes.

The error messages shown to users when they try invalid set operations on MAP and VARIANT types have been updated.

Before:

  • Messages only referred to DataFrame operations.

After:

  • Messages now apply to both DataFrame and SQL queries.

This is purely a change in wording; the actual execution behavior is unchanged.


How was this patch tested?

  • Verified existing unit tests for SET_OPERATION_ON_MAP_TYPE and SET_OPERATION_ON_VARIANT_TYPE in DataFrameSetOperationsSuite.scala and CollationSuite.scala.
  • Ensured all related tests continue to pass after updating the error messages in error-conditions.json.
  • Confirmed that no functional changes were introduced, only updates to error message text definitions.

Was this patch authored or co-authored using generative AI tooling?

No

@KenanAdel KenanAdel changed the title SPARK-57576: [SQL] Generalize error messages for set operations on MAP and VARIANT types [SPARK-57576] [SQL] Generalize error messages for set operations on MAP and VARIANT types Jun 19, 2026
@KenanAdel KenanAdel changed the title [SPARK-57576] [SQL] Generalize error messages for set operations on MAP and VARIANT types [SPARK-57576][SQL] Generalize error messages for set operations on MAP and VARIANT types Jun 19, 2026
@KenanAdel

Copy link
Copy Markdown
Author

Closing this PR as I noticed another contributor has already submitted a comprehensive solution with full test coverage for this issue. Glad to see it being addressed

@KenanAdel KenanAdel closed this Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant