Skip to content

Add proxy support to Databricks connections#68527

Open
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:databricks-proxy-support
Open

Add proxy support to Databricks connections#68527
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:databricks-proxy-support

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Jun 14, 2026

Copy link
Copy Markdown
Contributor

Add explicit proxy support for Databricks connections.

What changed

This adds a documented proxies connection extra for the Databricks provider. The extra is intentionally allowlisted and only accepts http and https keys, for example:

{
  "proxies": {
    "http": "http://proxy.example.com:8080",
    "https": "http://proxy.example.com:8443"
  }
}

The configured proxy is applied to:

  • Databricks REST API calls
  • Databricks OAuth token exchanges
  • Azure Identity token acquisition for AAD service principal and default Azure credential authentication
  • Sync and async Databricks hook paths

Why

Users running Airflow behind a corporate proxy could not authenticate to Azure Databricks because token acquisition to Azure/Databricks endpoints did not receive proxy configuration.

This fixes #32576 while keeping connection extras explicit and safe. The implementation does not pass arbitrary connection extras through to client libraries; it validates and forwards only the reviewed proxies extra.

Testing

I added coverage for:

  • Valid proxies parsing from Databricks connection extras
  • Rejection of invalid proxy shapes and unsupported keys
  • Sync Databricks REST API calls using proxies
  • Async Databricks REST API calls using the matching aiohttp proxy
  • AAD service principal token paths passing proxies to Azure Identity
  • Databricks service-principal OAuth token exchange using proxies
  • Sync and async token/API combinations
  • Databricks SQL connector kwargs not receiving proxies

Commands run:

  • uv run ruff format providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py
  • uv run ruff check --fix providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py
  • uv run ruff format providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
  • uv run ruff check --fix providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py
  • uv run ./scripts/ci/prek/check_new_airflow_exception_usage.py providers/databricks/src/airflow/providers/databricks/hooks/databricks_base.py
  • uv run --project providers/databricks pytest providers/databricks/tests/unit/databricks/hooks/test_databricks_base.py providers/databricks/tests/unit/databricks/hooks/test_databricks.py providers/databricks/tests/unit/databricks/hooks/test_databricks_sql.py -q

Result: 284 passed; AirflowException usage check passed.

closes: #32576


Was generative AI tooling used to co-author this PR?
  • Yes — Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines


Drafted-by: Codex (GPT-5) (no human review before posting)

@Vamsi-klu Vamsi-klu force-pushed the databricks-proxy-support branch from fdd7d60 to cdfb613 Compare June 14, 2026 03:52
@eladkal eladkal requested a review from jroachgolf84 June 14, 2026 05:17
@Vamsi-klu Vamsi-klu marked this pull request as ready for review June 14, 2026 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support connection with Proxy in Databricks provider

2 participants