Updates to allow MySQL clusters (#13)

Co-authored-by: Marta Paes <[email protected]>

CONTRIBUTING.md

@@ -20,7 +20,7 @@ To test the module manually, follow these steps:
 1. After the resources have been created, go to the Target Groups in the AWS console and make sure that the health checks are passing. If they are not, you will need to add the subnet CIDR blocks of your RDS instance to the security groups of your RDS instance. For more information, see [this AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html).
 1. Next, run the queries in the output to create the connection in Materialize.
 1. Finally, in your AWS console, under the Endpoint Service that was created, approve the connection request from the Materialize instance and check that the connection is active.
-1. You can now create a Postgres source in Materialize using the connection name from the output.
+1. You can now create a PostgreSQL or MySQL source in Materialize using the connection name from the output.
 1. Finally, drop the connection in Materialize and run `terraform destroy` to clean up the resources.
 
 ## Cutting a new release

README.md

@@ -3,29 +3,29 @@
 > [!WARNING]
 > This is provided on a best-effort basis and Materialize cannot offer support for this module
 
-This repository contains a Terraform module that configures a PrivateLink endpoint for an existing Amazon RDS Postgres database to connect to Materialize.
+This repository contains a Terraform module that configures a PrivateLink endpoint for existing Amazon RDS PostgreSQL or MySQL databases to connect to Materialize.
 
 The module creates the following resources:
-- Target group for the RDS instance
-- Network Load Balancer for the RDS instance
-- TCP listener for the NLB to forward traffic to the target group
-- A VPC endpoint service for your RDS instance
-- Lambda Function to check and update the IP address of the RDS instance in the NLB target group
+- Target group for each RDS instance
+- Network Load Balancer for the RDS instances
+- TCP listener for the NLB to forward traffic to the target groups
+- A VPC endpoint service for your RDS instances
+- Lambda Function to check and update the IP addresses of the RDS instances in the NLB target groups
 - IAM Role and Policy to give the Lambda function necessary permissions
 - Event Source Mapping, Event Rule, and Target: Triggers the Lambda function every _n_ minutes
 - Lambda Permission: Allows the event to invoke the Lambda function
 
 ## Important Remarks
 
 > [!NOTE]
-> The RDS instance needs to be private. If your RDS instance is public, there is no need to use PrivateLink.
+> The RDS instances need to be private. If your RDS instances are public, there is no need to use PrivateLink.
 
 > [!NOTE]
 > When using Aurora, the RDS instance needs to be a **writer** instance as the reader instances will not work.
 
-- The RDS instance must be in the same VPC as the PrivateLink endpoint.
+- The RDS instances must be in the same VPC as the PrivateLink endpoint.
 - Review this module with your Cloud Security team to ensure that it meets your security requirements.
-- Finally, after the Terraform module has been applied, you will need to make sure that the Target Groups health checks are passing. As the NLB does not have security groups, you will need to make sure that the NLB is able to reach the RDS instance by allowing the subnet CIDR blocks in the security groups of the RDS instance.
+- Finally, after the Terraform module has been applied, you will need to make sure that the Target Groups health checks are passing. As the NLB does not have security groups, you will need to make sure that the NLB is able to reach the RDS instances by allowing the subnet CIDR blocks in the security groups of the RDS instances.
 
 To override the default AWS provider variables, you can export the following environment variables:
 
@@ -46,7 +46,7 @@ cp terraform.tfvars.example terraform.tfvars
 
 | Name | Description | Type | Example | Required |
 |------|-------------|:----:|:-----:|:-----:|
-| mz_rds_instance_names | The name of the RDS instances | list | `{ name = "instance1", listener_port = 5001 }` | yes |
+| mz_rds_instance_names | The name and listener port of the RDS instances | list | `{ name = "instance1", listener_port = 5001 }` | yes |
 | mz_rds_vpc_id | The VPC ID of the RDS instance | string | `'vpc-1234567890abcdef0'` | yes |
 | mz_acceptance_required | Whether or not to require manual acceptance of new connections | bool | `true` | no |
 | schedule_expression | [The scheduling expression](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule#schedule_expression). For example, `cron(0 20 * * ? *)` | string | `'rate(5 minutes)'` | no |
@@ -62,7 +62,7 @@ terraform apply
 
 After the Terraform module has been applied, you will see the following output.
 
-You can follow the instructions in the output to configure the PrivateLink endpoint and the Postgres connections in Materialize.
+You can follow the instructions in the output to configure the PrivateLink endpoint and the database connections in Materialize.
 
 First, you will need to create the PrivateLink endpoint in Materialize:
 
@@ -83,29 +83,31 @@ mz_rds_private_link_endpoint_sql = <<EOT
 EOT
 ```
 
-After that, you will need to create the Postgres connections in Materialize, if you have multiple RDS instances, you will see multiple SQL statements:
+After that, you will need to create the database connections in Materialize. If you have multiple RDS instances, you will see multiple SQL statements:
 
 ```sql
-mz_rds_postgres_connection_sql   = {
+mz_rds_database_connection_sql   = {
     rds-instance-name = <<-EOT
           -- Create a secret for the password for rds-instance-name
-              CREATE SECRET rds-instance-name_pgpass AS 'YOUR_PG_PASSWORD_FOR_rds-instance-name';
+              CREATE SECRET rds-instance-name_dbpass AS 'YOUR_DB_PASSWORD_FOR_rds-instance-name';
               -- Create the connection to the RDS instance using the listener port
-              CREATE CONNECTION rds-instance-name_pg_conn TO POSTGRES (
+              CREATE CONNECTION rds-instance-name_db_conn TO POSTGRES (
                   HOST 'rds-instance-name.ctthmav6dsti.us-east-1.rds.amazonaws.com',
                   PORT 5001,
                   DATABASE postgres,
                   USER postgres,
-                  PASSWORD SECRET rds-instance-name_pgpass,
+                  PASSWORD SECRET rds-instance-name_dbpass,
                   AWS PRIVATELINK privatelink_svc
               );
       EOT
 }
 ```
 
+Note: For MySQL instances, replace `POSTGRES` with `MYSQL` in the connection creation SQL, and adjust the `DATABASE` and `USER` fields accordingly.
+
 ### Output details: Configure Materialize
 
-Once the Terraform module has been applied, you can configure Materialize to connect to the RDS instance using the PrivateLink endpoint:
+Once the Terraform module has been applied, you can configure Materialize to connect to the RDS instances using the PrivateLink endpoint:
 
 - Connect to the Materialize instance using `psql`
 - Run the SQL statement from the output of the `terraform apply` command to configure the PrivateLink connection, example:
@@ -130,10 +132,12 @@ SELECT principal
 
 - Add the allowed principals to the Endpoint Service configuration in the AWS console
 
-- Finally, run the last SQL statement from the output of the `terraform apply` command to create the Postgres connection which will use the PrivateLink endpoint. If you have multiple RDS instances, you will see multiple SQL statements:
+- Finally, run the last SQL statement from the output of the `terraform apply` command to create the database connection which will use the PrivateLink endpoint. If you have multiple RDS instances, you will see multiple SQL statements:
+
+For PostgreSQL instances:
 
 ```sql
--- Create the connection to the RDS instance
+-- Create the connection to the PostgreSQL RDS instance
 CREATE CONNECTION pg_connection TO POSTGRES (
     HOST 'instance.foo000.us-west-1.rds.amazonaws.com',
     PORT 5432,
@@ -144,9 +148,22 @@ CREATE CONNECTION pg_connection TO POSTGRES (
 );
 ```
 
+For MySQL instances:
+
+```sql
+-- Create the connection to the MySQL RDS instance
+CREATE CONNECTION mysql_connection TO MYSQL (
+    HOST 'mysql-instance.foo000.us-west-1.rds.amazonaws.com',
+    PORT 3306,
+    USER your_mysql_user,
+    PASSWORD SECRET mysql_dbpass,
+    AWS PRIVATELINK privatelink_svc
+);
+```
+
 After that go to your AWS console and check that the VPC endpoint service has a pending connection request from the Materialize instance which you can approve.
 
-After the connection request has been approved, you can create a Postgres source in Materialize using the `pg_connection` connection.
+After the connection request has been approved, you can create a database source in Materialize using the `pg_connection` or `mysql_connection` connection.
 
 ## Materialize Documentation
 

outputs.tf

@@ -16,20 +16,25 @@ output "mz_rds_private_link_endpoint_sql" {
 EOF
 }
 
-# Generates SQL queries to create the PostgreSQL connections using the listener port
-output "mz_rds_postgres_connection_sql" {
-  description = "SQL queries to create the PostgreSQL connections using the listener port. Run these queries after creating the VPC endpoint service. If you have multiple RDS instances, run these queries for each instance."
+# Generates SQL queries to create the database connections using the listener port
+output "mz_rds_database_connection_sql" {
+  description = "SQL queries to create the database connections using the listener port. Run these queries after creating the VPC endpoint service. If you have multiple RDS instances, run these queries for each instance."
   value = { for inst in var.mz_rds_instance_details : inst.name => <<EOF
     -- Create a secret for the password for ${inst.name}
-    CREATE SECRET ${inst.name}_pgpass AS 'YOUR_PG_PASSWORD_FOR_${inst.name}';
+    CREATE SECRET "${inst.name}_dbpass" AS 'YOUR_DB_PASSWORD_FOR_${inst.name}';
 
     -- Create the connection to the RDS instance using the listener port
-    CREATE CONNECTION ${inst.name}_pg_conn TO POSTGRES (
+    CREATE CONNECTION "${inst.name}_db_conn" TO ${
+      contains(["mysql", "mariadb", "aurora-mysql"], data.aws_db_instance.mz_rds_instance[inst.name].engine) ? "MYSQL" :
+      contains(["postgres", "aurora-postgresql"], data.aws_db_instance.mz_rds_instance[inst.name].engine) ? "POSTGRES" :
+      upper(data.aws_db_instance.mz_rds_instance[inst.name].engine)
+    } (
         HOST '${data.aws_db_instance.mz_rds_instance[inst.name].address}',
         PORT ${inst.listener_port},
-        DATABASE postgres,
-        USER postgres,
-        PASSWORD SECRET ${inst.name}_pgpass,
+        ${contains(["postgres", "aurora-postgresql"], data.aws_db_instance.mz_rds_instance[inst.name].engine) ? 
+          "DATABASE ${data.aws_db_instance.mz_rds_instance[inst.name].db_name}," : ""}
+        USER ${data.aws_db_instance.mz_rds_instance[inst.name].master_username},
+        PASSWORD SECRET "${inst.name}_dbpass",
         AWS PRIVATELINK privatelink_svc
     );
 EOF

terraform.tfvars.example

@@ -6,5 +6,6 @@ mz_rds_instance_details = [
   { name = "instance1", listener_port = 5001 },
   { name = "instance2", listener_port = 5002 }
 ]
-# The VPC ID of the existing MSK cluster
+
+# The VPC ID of the existing RDS instance
 mz_rds_vpc_id =