[SPARK-57175][SQL] Extend nested column pruning to exists and forall over arrays of structs

[sunchao]

Why are the changes needed?

SPARK-57175 follows SPARK-57022, which added nested column pruning for transform over array<struct> inputs. The same optimization does not currently apply to the exists and forall higher-order array functions.

For example:

SELECT exists(rule_results, rule -> rule.rule_version > 10)
FROM events

If rule_results contains additional fields, Spark currently retains the full element struct in the scan schema even though the predicate only reads rule_version. This causes unnecessary Parquet and ORC input reads for wide array element schemas.

What changes were proposed in this PR?

• Share the nested-field collection path introduced for ArrayTransform with ArrayExists and ArrayForAll.
• Rewrite the bound lambda variable type and GetStructField ordinals against the projected element schema after pruning.
• Keep the conservative fallback when a lambda consumes the whole element.
• Add Catalyst and datasource tests covering schema discovery, ordinal rewrites, predicate-path schema merging, and whole-element fallback.

ArrayFilter and ArraySort remain out of scope because they return original input elements and require a different downstream-schema design.

Does this PR introduce any user-facing change?

Yes. Eligible queries using exists or forall over arrays of structs can read a narrower input schema. Query results and SQL APIs are unchanged.

How was this patch tested?

• JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.SchemaPruningSuite" "sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.parquet.ParquetV2SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV2SchemaPruningSuite -- -z Array"
• git diff --check

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)