-
Notifications
You must be signed in to change notification settings - Fork 8
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
c46aa7d
commit e202851
Showing
5 changed files
with
191 additions
and
10 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| --- | ||
| page_title: "Source versioning: migrating to `materialize_source_table` Resource" | ||
| subcategory: "" | ||
| description: |- | ||
|
|
||
| --- | ||
|
|
||
| # Source versioning: migrating to `materialize_source_table` Resource | ||
|
|
||
| In previous versions of the Materialize Terraform provider, source tables were defined within the source resource itself and were considered subsources of the source rather than separate entities. | ||
|
|
||
| This guide will walk you through the process of migrating your existing source table definitions to the new `materialize_source_table` resource. | ||
|
|
||
| ## Old Approach | ||
|
|
||
| Previously, source tables were defined directly within the source resource: | ||
|
|
||
| ### Example: MySQL Source | ||
|
|
||
| ```hcl | ||
| resource "materialize_source_mysql" "mysql_source" { | ||
| name = "mysql_source" | ||
| cluster_name = "cluster_name" | ||
|
|
||
| mysql_connection { | ||
| name = materialize_connection_mysql.mysql_connection.name | ||
| } | ||
|
|
||
| table { | ||
| upstream_name = "mysql_table1" | ||
| upstream_schema_name = "shop" | ||
| name = "mysql_table1_local" | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The same approach was used for other source types such as Postgres and the load generator sources. | ||
|
|
||
| ## New Approach | ||
|
|
||
| The new approach separates source definitions and table definitions. You will now create the source without specifying the tables, and then define each table using the `materialize_source_table` resource. | ||
|
|
||
| ## Manual Migration Process | ||
|
|
||
| This manual migration process requires users to create new source tables using the new `materialize_source_table` resource and then remove the old ones. | ||
|
|
||
| ### Step 1: Define `materialize_source_table` Resources | ||
|
|
||
| Before making any changes to your existing source resources, create new `materialize_source_table` resources for each table that is currently defined within your sources. This ensures that the tables are preserved during the migration: | ||
|
|
||
| ```hcl | ||
| resource "materialize_source_table" "mysql_table_from_source" { | ||
| name = "mysql_table1_from_source" | ||
| schema_name = "public" | ||
| database_name = "materialize" | ||
|
|
||
| source { | ||
| name = materialize_source_mysql.mysql_source.name | ||
| // Define the schema and database for the source if needed | ||
| } | ||
|
|
||
| upstream_name = "mysql_table1" | ||
| upstream_schema_name = "shop" | ||
|
|
||
| ignore_columns = ["about"] | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 2: Apply the Changes | ||
|
|
||
| Run `terraform plan` and `terraform apply` to create the new `materialize_source_table` resources. This step ensures that the tables are defined separately from the source and are not removed from Materialize. | ||
|
|
||
| > **Note:** This will start an ingestion process for the newly created source tables. | ||
|
|
||
| ### Step 3: Remove Table Blocks from Source Resources | ||
|
|
||
| Once the new `materialize_source_table` resources are successfully created, you can safely remove the `table` blocks from your existing source resources: | ||
|
|
||
| ```hcl | ||
| resource "materialize_source_mysql" "mysql_source" { | ||
| name = "mysql_source" | ||
| cluster_name = "cluster_name" | ||
|
|
||
| mysql_connection { | ||
| name = materialize_connection_mysql.mysql_connection.name | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| This will drop the old tables from the source resources. | ||
|
|
||
| ### Step 4: Update Terraform State | ||
|
|
||
| After removing the `table` blocks from your source resources, run `terraform plan` and `terraform apply` again to update the Terraform state and apply the changes. | ||
|
|
||
| ### Step 5: Verify the Migration | ||
|
|
||
| After applying the changes, verify that your tables are still correctly set up in Materialize by checking the table definitions using Materialize’s SQL commands. | ||
|
|
||
| During the migration, you can use both the old `table` blocks and the new `materialize_source_table` resources simultaneously. This allows for a gradual transition until the old method is fully deprecated. | ||
|
|
||
| ## Automated Migration Process (TBD) | ||
|
|
||
| > **Note:** This will still not work as the previous source tables are considered subsources of the source and are missing from the `mz_tables` table in Materialize so we can't import them directly without recreating them. | ||
|
|
||
| Once the migration on the Materialize side has been implemented, a more automated migration process will be available. The steps will include: | ||
|
|
||
| ### Step 1: Define `materialize_source_table` Resources | ||
|
|
||
| First, define the new `materialize_source_table` resources for each table: | ||
|
|
||
| ```hcl | ||
| resource "materialize_source_table" "mysql_table_from_source" { | ||
| name = "mysql_table1_from_source" | ||
| schema_name = "public" | ||
| database_name = "materialize" | ||
|
|
||
| source { | ||
| name = materialize_source_mysql.mysql_source.name | ||
| // Define the schema and database for the source if needed | ||
| } | ||
|
|
||
| upstream_name = "mysql_table1" | ||
| upstream_schema_name = "shop" | ||
|
|
||
| ignore_columns = ["about"] | ||
| } | ||
| ``` | ||
|
|
||
| ### Step 2: Modify the Existing Source Resource | ||
|
|
||
| Next, modify the existing source resource by removing the `table` blocks and adding an `ignore_changes` directive for the `table` attribute. This prevents Terraform from trying to delete the tables: | ||
|
|
||
| ```hcl | ||
| resource "materialize_source_mysql" "mysql_source" { | ||
| name = "mysql_source" | ||
| cluster_name = "cluster_name" | ||
|
|
||
| mysql_connection { | ||
| name = materialize_connection_mysql.mysql_connection.name | ||
| } | ||
|
|
||
| lifecycle { | ||
| ignore_changes = [table] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| - **`lifecycle { ignore_changes = [table] }`**: This directive tells Terraform to ignore changes to the `table` attribute, preventing it from trying to delete tables that were previously defined in the source resource. | ||
|
|
||
| ### Step 3: Import the Existing Tables | ||
|
|
||
| You can then import the existing tables into the new `materialize_source_table` resources without disrupting your existing setup: | ||
|
|
||
| ```bash | ||
| terraform import materialize_source_table.mysql_table_from_source <region>:<table_id> | ||
| ``` | ||
|
|
||
| Replace `<region>` with the actual region and `<table_id>` with the table ID. You can find the table ID by querying the `mz_tables` table. | ||
|
|
||
| ### Step 4: Run Terraform Plan and Apply | ||
|
|
||
| Finally, run `terraform plan` and `terraform apply` to ensure that everything is correctly set up without triggering any unwanted deletions. | ||
|
|
||
| This approach allows you to migrate your tables safely without disrupting your existing setup. | ||
|
|
||
| ## Importing Existing Tables | ||
|
|
||
| To import existing tables into your Terraform state using the manual migration process, use the following command: | ||
|
|
||
| ```bash | ||
| terraform import materialize_source_table.table_name <region>:<table_id> | ||
| ``` | ||
|
|
||
| Ensure you replace `<region>` with the region where the table is located and `<table_id>` with the ID of the table. | ||
|
|
||
| > **Note:** The `upstream_name` and `upstream_schema_name` attributes are not yet implemented on the Materialize side, so the import process will not work until these changes are made. | ||
|
|
||
| ## Future Improvements | ||
|
|
||
| The Kafka and Webhooks sources are currently being implemented. Once these changes, the migration process will be updated to include them. |