[VL] Spark 4.x: Support Variant type

[Surbhi-Vijay]

Description

Spark-4.0 has added new data type VariantType. The support for VariantType and variant shredded data has been added to parquet reader and writer.

Refer:

• https://issues.apache.org/jira/browse/SPARK-50614
• [SPARK-48495][SQL][DOCS] Describe shredding scheme for Variant
• [SPARK-50614][SQL] Add Variant shredding support for Parquet
• [SPARK-54454][SQL] Enable variant shredding and variant logical type annotation configs by default

This feature has been enabled by default in Spark. Currently, since Gluten does not support VariantType, need to add fallback.

Gluten version

None

[Surbhi-Vijay]

Need to add fallback logic.

[baibaichen]

@Surbhi-Vijay should be fixed in #11726, set PARQUET_UNEXPECTED_METADATA_FALLBACK_ENABLED to ture to detect Variant shredding in footer.

[baibaichen]

Gluten now supports Variant data type via fallback to Spark JVM execution. Remaining work is Velox native Variant support, which is under discussion at facebookincubator/velox#17033.

Linking to the new Spark 4.x feature tracking issue #11910 for visibility.

Current test impact:

• GlutenVariantSuite: 2 excludes (SPARK-47546)
• GlutenVariantInferShreddingSuite: 1 exclude (infer shredding with mixed scale), spark41-only
• VeloxSQLQueryTestSettings: variant-field-extractions.sql TODO (spark41)