feat: implement distributed VECTOR_SEARCH with parallel fragment/index scanning

[summaryzb]

Summary

Implements distributed execution for the VECTOR_SEARCH table function, enabling Spark-parallel vector similarity search across Lance datasets. When enabled via spark.sql.lance.search.distributed.enabled=true, the driver plans one Spark task per execution unit
(indexed segment or fallback fragment), and each worker runs a local ANN scan or fallback to KNN scan without indexed segment. Results are merged with a global sort on _distance.

This provides horizontal scalability for vector search workloads on large datasets without requiring a centralized vector index server.

Behavior

Condition	Execution Path
distributed.enabled=false	Single-partition namespace.queryTable()
distributed.enabled=true, has vector index	One task per index segment, plus fallback tasks for unindexed fragments (unless fastSearch=true)
distributed.enabled=true, no index	One task per fragment (flat KNN)

Notice

• Indexed-unit path with IVF_PQ is implemented but not yet covered by E2E tests (pending CREATE INDEX ... USING IVF_PQ SQL support in lance-spark extensions) which is rely on feat: distributed vector index creation #605.
• The prefilter=true flag is forced for fallback units due to a lance-core JNI limitation where Scanner::nearest rejects fragment-restricted scans without a prefilter expression.
• Specify indexSegments in ScanOptions is pre required which is rely on feat(java): support segment-based distributed vector search lance#7169, other wise compile fails

Testing

• BaseSparkDistributedVectorSearchTest exercises fallback-only scenarios
• Spark 3.4 and 3.5 modules have thin test subclasses

[summaryzb]

CI RED is expected since it rely on lance-format/lance#7169

[summaryzb]

@jackye1995 @Xuanwo @LuciferYang PTAL