feat: expose Arrow-native geospatial option (databricks.arrow.native_geospatial)

[jatorre]

Summary

Adds Arrow-native geospatial support via databricks.arrow.native_geospatial ADBC connection option, implementing the full pipeline from Databricks geometry to standard GeoArrow geoarrow.wkb.

Why geoarrow.wkb: Databricks uses a proprietary Struct<srid: Int32, wkb: Binary> format for Arrow geometry serialization. While this works, it's not recognized by downstream tools. By converting to geoarrow.wkb (the standard Arrow extension type for geometry), we get maximum compatibility: DuckDB, pandas/geopandas, polars, GDAL, and any other GeoArrow-aware consumer can read geometry natively without custom parsing. The conversion is essentially free (zero-copy pointer extraction, no data copying).

Pipeline

Databricks server
  → Struct<srid: Int32, wkb: Binary>  (via geospatialAsArrow Thrift flag, SPARK-54232)
  → ADBC driver flattens to Binary + ARROW:extension:name=geoarrow.wkb
  → CRS from SRID encoded as PROJJSON in ARROW:extension:metadata
  → DuckDB adbc_scanner → native GEOMETRY (already supports geoarrow.wkb)

End result: SELECT * FROM adbc_scan(...) returns DuckDB GEOMETRY directly. No ST_AsBinary(), no ST_GeomFromWKB(). Zero geometry conversion in user code.

Dependency

Requires databricks/databricks-sql-go#328 for WithArrowNativeGeospatial() ConnOption.

Changes

• go/driver.go — Add OptionArrowNativeGeospatial constant
• go/database.go — useArrowNativeGeospatial field, GetOption/SetOption, connection passthrough
• go/connection.go — Pass flag to statements
• go/statement.go — Pass flag to IPC reader adapter
• go/ipc_reader_adapter.go — Core geoarrow conversion:
  • isGeometryStruct() — detect Databricks Struct<srid: Int32, wkb: Binary>
  • detectGeometryColumns() — find geometry struct column indices
  • buildGeoArrowSchemaWithoutCRS() — eagerly rewrite schema: Struct → Binary with geoarrow.wkb (needed before first Next() since consumers read schema upfront)
  • buildGeoArrowSchema() — rebuild schema with SRID-based CRS from first record batch, encoded as PROJJSON in ARROW:extension:metadata
  • transformRecordForGeoArrow() — extract wkb child array from struct per batch (zero-copy, O(columns) not O(rows))

Usage

-- DuckDB with adbc_scanner: geometry arrives as native GEOMETRY
SET VARIABLE db_conn = (SELECT adbc_connect({
  'driver': 'libadbc_driver_databricks.dylib',
  'databricks.server_hostname': '...',
  'databricks.http_path': '/sql/1.0/warehouses/...',
  'databricks.access_token': 'dapi...',
  'databricks.arrow.native_geospatial': 'true'
}));

-- geom is DuckDB GEOMETRY — no ST_GeomFromWKB needed!
SELECT * FROM adbc_scan(getvariable('db_conn')::BIGINT, 'SELECT * FROM my_geo_table');

Benchmarks (vs baseline ST_AsBinary + ST_GeomFromWKB)

Dataset	Baseline	GeoArrow	Speedup
100k points	27.04s (3.7k rows/sec)	3.85s (26k rows/sec)	7.0x
10k polygons	8.41s (1.2k rows/sec)	2.33s (4.3k rows/sec)	3.6x

SRID / CRS Handling

Databricks uses per-row SRID in Struct<srid, wkb>. The GeoArrow spec defines CRS per-column, not per-row. The driver reads the SRID from the first non-null row of each geometry column in the first record batch and encodes it as PROJJSON CRS in ARROW:extension:metadata:

{"crs":{"type":"projjson","properties":{"name":"EPSG:4326"},"id":{"authority":"EPSG","code":4326}}}

In practice, Databricks geometry/geography uses a consistent SRID within a column (typically 0 or 4326), so this per-column CRS approach is safe and follows the GeoArrow standard. Non-zero SRIDs (e.g. 4326, 3857) are preserved; SRID 0 produces empty CRS metadata (the geoarrow default).

[paleolimbot]

Cool!