[WIP] ADH-8176: added service discovery via Zookeeper for Spark Connect

[werzerbb]

Summary

Implemented HA (High Availability) support for Spark Connect in Kyuubi. When a Kyuubi server
crashes, clients transparently failover to another live server and resume work without user
intervention.

Architecture diagram: https://github.com/arenadata/kyuubi/blob/feature/ADH-8176-spark-connect-ha/docs/spark_connect/spark_connect_ha_architecture.mmd

What was added

• ZooKeeper service discovery: clients use
  sc://zk1:2181,zk2:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi_sc URL
  to discover live Kyuubi SC servers
• Python transparent failover (FailoverChannel): wraps gRPC channel, catches UNAVAILABLE,
  switches to next ZK server automatically
• JVM transparent failover (FailoverManagedChannel + SparkConnectBridge): same for
  Java/Scala clients;
• JDBC token store (JdbcTokenStore): persists auth tokens in PostgreSQL/MySQL/SQLite so
  tokens remain valid after server failover; uses existing kyuubi.metadata.store.jdbc.* config
• CONNECTION share level isolation: engine ZK path uses gRPC session_id (client-generated, stable
  across reconnects) as engineRefId;

New config options

Key	Default	Description
kyuubi.ha.spark.connect.namespace	kyuubi_sc	ZooKeeper namespace for Spark Connect server registration (must differ from kyuubi.ha.namespace to avoid Thrift clients discovering gRPC endpoints)

Example usage

$ KYUUBI_AUTH=kerberos pyspark3 --remote "sc://vdmitriev-adh-sun2-hadoop-1.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-2.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-3.ru-central1.internal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=arenadata/cluster/11/kyuubi_sc;use_ssl=true"
...
Client connected to the Spark Connect server at vdmitriev-adh-sun2-hadoop-1.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-2.ru-central1.internal:2181,vdmitriev-adh-sun2-hadoop-3.ru-central1.internal:2181
SparkSession available as 'spark'.
>>> spark.sql("SELECT current_user()").show()
+--------------+
|current_user()|
+--------------+
|     vdmitriev|
+--------------+

Dependencies

• Requires ADH-8176: prepare for HA spark connect spark#54 (GrpcRetryHandler fix) to be merged first