Skip to content

Fix Databricks operators with templated json payloads#68519

Open
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:codex/35433-databricks-payload
Open

Fix Databricks operators with templated json payloads#68519
Vamsi-klu wants to merge 1 commit into
apache:mainfrom
Vamsi-klu:codex/35433-databricks-payload

Conversation

@Vamsi-klu

Copy link
Copy Markdown
Contributor

Fix Databricks operators so templated json payloads are rendered before the final Databricks request payload is built.

This updates DatabricksCreateJobsOperator, DatabricksSubmitRunOperator, and DatabricksRunNowOperator to:

  • assign templated constructor fields directly in __init__
  • build and validate the final payload in execute()
  • accept rendered mappings, JSON strings, Python-literal dict strings, and XComArg values
  • merge rendered named parameters after json, so named arguments keep precedence
  • preserve existing validation for dbt/git source, pipeline id/name, job id/name, deferrable execution, and repair-run paths

This also adds regression coverage for templated named parameters and XComArg/classic Jinja payloads.

closes: #35433
related: #40788

Validation:

  • uv run --project providers/databricks pytest providers/databricks/tests/unit/databricks/operators/test_databricks.py -xvs
  • prek run --from-ref upstream/main --stage pre-commit
  • prek run --from-ref upstream/main --stage manual
  • breeze ci selective-check --commit-ref HEAD
  • breeze testing providers-tests --test-type "Providers[databricks]"

Note: breeze selective-checks --commit-ref HEAD is documented in AGENTS.md/selective-checks docs, but the current Breeze command is breeze ci selective-check --commit-ref HEAD.


Was generative AI tooling used to co-author this PR?
  • Yes - Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines


Drafted-by: Codex (GPT-5) (no human review before posting)

@Vamsi-klu Vamsi-klu force-pushed the codex/35433-databricks-payload branch from 41776d3 to 7350e3f Compare June 13, 2026 23:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Databricks operators' json parameter compatible with XComs, Jinja expression values

1 participant