Added support for creating all-purpose clusters

[andrewnester]

Changes

Added support for creating all-purpose clusters

Example of configuration

bundle:
  name: clusters

resources:
  clusters:
    test_cluster:
      cluster_name: "Test Cluster"
      num_workers: 2
      node_type_id: "i3.xlarge"
      autoscale:
        min_workers: 2
        max_workers: 7
      spark_version: "13.3.x-scala2.12"
      spark_conf:
        "spark.executor.memory": "2g"

  jobs:
    test_job:
      name: "Test Job"
      tasks:
        - task_key: test_task
          existing_cluster_id: ${resources.clusters.test_cluster.id}
          notebook_task:
            notebook_path: "./src/test.py"

targets:
    development:
      mode: development
      compute_id: ${resources.clusters.test_cluster.id}

Tests

Added unit, config and E2E tests

[pietern]

Is a cluster started if it is deployed?

This is not strictly necessary for bundle deployments because it will be started by jobs that use it. If it is started, deploy may take a long time.

[pietern]

How can folks use this with:

bundle:
  compute_id: ${resources.clusters.my_cluster.id}

Also, I'd love for this PR to also rename both the flag and the field to cluster_id.

[andrewnester]

@pietern it is started by default so indeed deployment takes longer in this case

[andrewnester]

How can folks use this with:
bundle:
compute_id: ${resources.clusters.my_cluster.id}

You can do this in target overrides but I'd better remove it from example because it is confusing, this is just to illustrate that reference string can be used for compute id

[pietern]

@andrewnester Is there a way we can avoid starting it (or waiting for it to start)?

[andrewnester]

@pietern unfortunately we can't as TF provider and Go SDK explicitly wait for cluster to be running
https://github.com/databricks/terraform-provider-databricks/blob/main/clusters/resource_cluster.go#L413C2-L420

[lennartkats-db]

@andrewnester Waiting for it to start indeed seems pretty unfortunate, especially for development ... Does that happen every time, or just when it's first created?

[lennartkats-db]

You can do this in target overrides but I'd better remove it from example because it is confusing, this is just to illustrate that reference string can be used for compute id

But that would work, right? Setting bundle.cluster_id? Otherwise these all-purpose clusters are quite hard to use. You can't use a regular override to override a job cluster (new_cluster) with an all-purpose cluster (existing_cluster_id)

[edit]I opened a thread on this below[/edit]

[pietern]

Adding the label because we need the TF-side change to be released first.

[pietern]

Looks good!

Left a review on the TF PR, we can get this in when that is merged, the provider is released, and we have bumped the provider here.

[pietern]

Neat, this can be a function under libs/dyn (no need to include here).

[lennartkats-db]

Starting a thread on the above for tracking purposes

You can do this in target overrides but I'd better remove it from example because it is confusing, this is just to illustrate that reference string can be used for compute id

But that would work, right? Setting bundle.cluster_id? Otherwise these all-purpose clusters are quite hard to use. You can't use a regular override to override a job cluster (new_cluster) with an all-purpose cluster (existing_cluster_id)

[andrewnester]

Yes, it's possible to specify compute_id or cluster_id in your target with variable reference to a new cluster. I've added an example

[pietern]

How does it work with run_as?

I can see how it works with shared-user clusters, but is (I think?) incompatible with single-user clusters.

[andrewnester]

It does not work with run_as, this list is just list of available resources. Then it's added in allowList below as per comment in the section below

	// Action Item: If you are adding a new resource to DABs, please check in with
	// the relevant owning team whether the resource should be on the allow list or (implicitly) on
	// the deny list. Any resources that could have run_as semantics in the future
	// should be on the deny list.

[pietern]

Please TAL at the last comment and failing test (looks like no_wait property missing).