Cluster gets recreated during deployment

[GewoonMaarten]

Describe the issue

When I deploy the cluster resource it triggers a Compute edited by ... event and I need to wait a bit before the cluster is running again.

The event:

{
  "previous_attributes": {
    "cluster_name": "[dev ...] Development cluster",
    "spark_version": "17.3.x-scala2.13",
    "spark_conf": {
      "spark.sql.shuffle.partitions": "auto"
    },
    "azure_attributes": {
      "first_on_demand": 1,
      "availability": "SPOT_WITH_FALLBACK_AZURE",
      "spot_bid_max_price": -1
    },
    "node_type_id": "Standard_D4ds_v5",
    "driver_node_type_id": "Standard_D4ds_v5",
    "custom_tags": {},
    "autotermination_minutes": 60,
    "enable_elastic_disk": true,
    "disk_spec": {},
    "cluster_source": "UI",
    "policy_id": "...",
    "enable_local_disk_encryption": false,
    "instance_source": {
      "node_type_id": "Standard_D4ds_v5"
    },
    "driver_instance_source": {
      "node_type_id": "Standard_D4ds_v5"
    },
    "data_security_mode": "USER_ISOLATION",
    "effective_spark_version": "17.3.x-scala2.13",
    "release_version": "17.3.10"
  },
  "attributes": {
    "cluster_name": "[dev ...] Development cluster",
    "spark_version": "17.3.x-scala2.13",
    "spark_conf": {
      "spark.sql.shuffle.partitions": "auto"
    },
    "azure_attributes": {
      "first_on_demand": 1,
      "availability": "SPOT_WITH_FALLBACK_AZURE",
      "spot_bid_max_price": -1
    },
    "node_type_id": "Standard_D4ds_v5",
    "driver_node_type_id": "Standard_D4ds_v5",
    "custom_tags": {},
    "autotermination_minutes": 60,
    "enable_elastic_disk": true,
    "disk_spec": {},
    "cluster_source": "UI",
    "policy_id": "...",
    "enable_local_disk_encryption": false,
    "instance_source": {
      "node_type_id": "Standard_D4ds_v5"
    },
    "driver_instance_source": {
      "node_type_id": "Standard_D4ds_v5"
    },
    "data_security_mode": "USER_ISOLATION",
    "effective_spark_version": "17.3.x-scala2.13"
  },
  "previous_cluster_size": {
    "autoscale": {
      "min_workers": 2,
      "max_workers": 20,
      "target_workers": 13
    }
  },
  "cluster_size": {
    "autoscale": {
      "min_workers": 2,
      "max_workers": 20,
      "target_workers": 13
    }
  },
  "user": "..."
}

The only property that has been updated is release_version, which I cannot set. I also checked #4933 and #3286, but I think I have a different problem databricks bundle plan -o json does not show anything intresetting:

❯ databricks bundle plan -o json
Building default...
{
  "plan": {
    "resources.clusters.development_cluster": {
      "action": "update"
    },
    "resources.jobs.test": {
      "action": "update"
    }
  }
}

Configuration

variables:
  policy_id:
    description: Cluster policy id for cost allocation
    lookup:
      cluster_policy: "test"
  spark_version:
    description: Shared Spark runtime version for all clusters
    default: "17.3.x-scala2.13"
  node_type_id:
    description: Shared node type for all clusters
    default: "Standard_D4ds_v5"

targets:
  dev:
    mode: development
    default: true

    cluster_id: ${resources.clusters.development_cluster.id}

    resources:
      clusters:
        development_cluster:
          cluster_name: Development cluster
          spark_version: ${var.spark_version}
          node_type_id: ${var.node_type_id}
          spark_conf:
            spark.sql.shuffle.partitions: auto
          autoscale:
            min_workers: 2
            max_workers: 20
          autotermination_minutes: 60
          azure_attributes:
            availability: SPOT_WITH_FALLBACK_AZURE
          data_security_mode: DATA_SECURITY_MODE_AUTO
          policy_id: ${var.policy_id}

Steps to reproduce the behavior

1. Run databricks bundle deploy
2. On the second deployment cluster gets recreated

Expected Behavior

No updates to config don't trigger a cluster update

Actual Behavior

Cluster is updated

OS and CLI version

• Databricks CLI v0.278.0
• Databricks CLI v0.299.0

Is this a regression?

Did this work in a previous version of the CLI? If so, which versions did you try?

Debug Logs

Output logs if you run the command with debug logs enabled. Example: databricks bundle deploy --log-level=debug. Redact if needed

[andrewnester]

@GewoonMaarten thanks for reporting! Do you use Terraform or direct deployment mode? My assumption it's a Terraform mode. In this case I would suggest you to try to use direct deployment, where this type of the issues should not exist
https://docs.databricks.com/aws/en/dev-tools/bundles/direct

[GewoonMaarten]

@andrewnester Thanks for the response! I migrated to the direct deployment. It still recreates the cluster, but for a different reason. There are some custom_tags automatically applied by our policy which I have not defined in our databricks.yml. Is there a way to ignore those during deployment or is the only choice I have to also define them in our config?

[andrewnester]

@GewoonMaarten do you have custom_tags section in the databricks.yml but it has a different set of tags or you don't have custom_tags in databricks.yml at all?

[GewoonMaarten]

@andrewnester I dont have any custom_tags. This is my full config:

    resources:
      clusters:
        development_cluster:
          cluster_name: Development cluster
          apply_policy_default_values: true
          spark_version: ${var.spark_version}
          node_type_id: ${var.node_type_id}
          spark_conf:
            spark.sql.shuffle.partitions: auto
          autoscale:
            min_workers: 2
            max_workers: 20
          autotermination_minutes: 60
          azure_attributes:
            availability: SPOT_WITH_FALLBACK_AZURE
          data_security_mode: DATA_SECURITY_MODE_AUTO
          policy_id: ${var.policy_id}