Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Populate partitions when reading databricks_sql_table #4486

Merged
merged 11 commits into from
Feb 26, 2025
1 change: 1 addition & 0 deletions NEXT_CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
* Add an example for Databricks Apps permissions ([#4475](https://github.com/databricks/terraform-provider-databricks/pull/4475)).
* Add explanation of timeouts to the troubleshooting guide ([#4482](https://github.com/databricks/terraform-provider-databricks/pull/4482)).
* Clarify that `databricks_token` and `databricks_obo_token` could be used only with workspace-level provider ([#4480](https://github.com/databricks/terraform-provider-databricks/pull/4480)).
* Clarify limitations whe importing `databricks_sql_table` ([#4483](https://github.com/databricks/terraform-provider-databricks/pull/4483)).

### Exporter

Expand Down
27 changes: 15 additions & 12 deletions docs/resources/sql_table.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ A `databricks_sql_table` is contained within [databricks_schema](schema.md), and

This resource creates and updates the Unity Catalog table/view by executing the necessary SQL queries on a special auto-terminating cluster it would create for this operation. You could also specify a SQL warehouse or cluster for the queries to be executed on.

~> This resource doesn't handle complex cases of schema evolution due to the limitations of Terraform itself. If you need to implement schema evolution it's recommended to use specialized tools, such as, [Luquibase](https://medium.com/dbsql-sme-engineering/advanced-schema-management-on-databricks-with-liquibase-1900e9f7b9c0) and [Flyway](https://medium.com/dbsql-sme-engineering/databricks-schema-management-with-flyway-527c4a9f5d67).
~> This resource doesn't handle complex cases of schema evolution due to the limitations of Terraform itself. If you need to implement schema evolution it's recommended to use specialized tools, such as, [Liquibase](https://medium.com/dbsql-sme-engineering/advanced-schema-management-on-databricks-with-liquibase-1900e9f7b9c0) and [Flyway](https://medium.com/dbsql-sme-engineering/databricks-schema-management-with-flyway-527c4a9f5d67).

## Example Usage

Expand Down Expand Up @@ -162,15 +162,15 @@ The following arguments are supported:
* `storage_location` - (Optional) URL of storage location for Table data (required for EXTERNAL Tables). Not supported for `VIEW` or `MANAGED` table_type.
* `data_source_format` - (Optional) External tables are supported in multiple data source formats. The string constants identifying these formats are `DELTA`, `CSV`, `JSON`, `AVRO`, `PARQUET`, `ORC`, and `TEXT`. Change forces the creation of a new resource. Not supported for `MANAGED` tables or `VIEW`.
* `view_definition` - (Optional) SQL text defining the view (for `table_type == "VIEW"`). Not supported for `MANAGED` or `EXTERNAL` table_type.
* `cluster_id` - (Optional) All table CRUD operations must be executed on a running cluster or SQL warehouse. If a cluster_id is specified, it will be used to execute SQL commands to manage this table. If empty, a cluster will be created automatically with the name `terraform-sql-table`.
* `cluster_id` - (Optional) All table CRUD operations must be executed on a running cluster or SQL warehouse. If a cluster_id is specified, it will be used to execute SQL commands to manage this table. If empty, a cluster will be created automatically with the name `terraform-sql-table`. Conflicts with `warehouse_id`.
* `warehouse_id` - (Optional) All table CRUD operations must be executed on a running cluster or SQL warehouse. If a `warehouse_id` is specified, that SQL warehouse will be used to execute SQL commands to manage this table. Conflicts with `cluster_id`.
* `cluster_keys` - (Optional) a subset of columns to liquid cluster the table by. Conflicts with `partitions`.
* `partitions` - (Optional) a subset of columns to partition the table by. Change forces the creation of a new resource. Conflicts with `cluster_keys`.
* `storage_credential_name` - (Optional) For EXTERNAL Tables only: the name of storage credential to use. Change forces the creation of a new resource.
* `owner` - (Optional) User name/group name/sp application_id of the schema owner.
* `owner` - (Optional) User name/group name/sp application_id of the table owner.
* `comment` - (Optional) User-supplied free-form text. Changing the comment is not currently supported on the `VIEW` table type.
* `options` - (Optional) Map of user defined table options. Change forces creation of a new resource.
* `properties` - (Optional) A map of table properties.
* `partitions` - (Optional) a subset of columns to partition the table by. Change forces the creation of a new resource. Conflicts with `cluster_keys`. Change forces creation of a new resource.

### `column` configuration block

Expand All @@ -191,7 +191,7 @@ In addition to all the arguments above, the following attributes are exported:

## Import

This resource can be imported by its full name:
This resource can be imported by its full name. `partitions` attribute is not imported due to Unity Catalog limitations and therefore should not be specified:

```bash
terraform import databricks_sql_table.this <catalog_name>.<schema_name>.<name>
Expand All @@ -200,24 +200,26 @@ terraform import databricks_sql_table.this <catalog_name>.<schema_name>.<name>
## Migration from `databricks_table`

The `databricks_table` resource has been deprecated in favor of `databricks_sql_table`. To migrate from `databricks_table` to `databricks_sql_table`:

1. Define a `databricks_sql_table` resource with arguments corresponding to `databricks_table`.
2. Add a `removed` block to remove the `databricks_table` resource without deleting the existing table by using the `lifecycle` block. If you're using Terraform version below v1.7.0, you will need to use the `terraform state rm` command instead.
3. Add an `import` block to add the `databricks_sql_table` resource, corresponding to the existing table. If you're using Terraform version below v1.5.0, you will need to use `terraform import` command instead.

For example, suppose we have the following `databricks_table` resource:

```hcl
resource "databricks_table" "this" {
catalog_name = "catalog"
schema_name = "schema"
name = "table"
table_type = "MANAGED"
catalog_name = "catalog"
schema_name = "schema"
name = "table"
table_type = "MANAGED"
data_source_format = "DELTA"
column {
name = "col1"
name = "col1"
type_name = "STRING"
type_json = "{\"type\":\"STRING\"}"
comment = "comment"
nullable = true
comment = "comment"
nullable = true
}
comment = "comment"
properties = {
Expand All @@ -227,6 +229,7 @@ resource "databricks_table" "this" {
```

The migration would look like this:

```hcl
# Leave this resource definition as-is.
resource "databricks_table" "this" { ... }
Expand Down
Loading