From c01189453f288380f696cc707cb6186a93882dbd Mon Sep 17 00:00:00 2001 From: Renato Rudnicki <77694243+renato-rudnicki@users.noreply.github.com> Date: Thu, 6 Jun 2024 11:47:18 -0300 Subject: [PATCH 1/4] Fix project name in files (#42) --- 0-bootstrap/README-GitHub.md | 68 +++++++++---------- 0-bootstrap/README-Jenkins.md | 48 ++++++------- 0-bootstrap/README-Terraform-Cloud.md | 42 ++++++------ 0-bootstrap/README.md | 20 +++--- 1-org/README.md | 22 +++--- 1-org/modules/cai-monitoring/README.md | 2 +- 1-org/modules/centralized-logging/README.md | 4 +- 2-environments/README.md | 18 ++--- 3-networks-dual-svpc/README.md | 28 ++++---- 4-projects/README.md | 26 +++---- 5-app-infra/0-gcp-policies/README.md | 4 +- 5-app-infra/1-artifact-publish/README.md | 16 ++--- 5-app-infra/2-artifact-publish-repo/README.md | 2 +- 5-app-infra/3-service-catalog/README.md | 16 ++--- 5-app-infra/4-service-catalog-repo/README.md | 2 +- 5-app-infra/5-vpc-sc/README.md | 2 +- 5-app-infra/6-machine-learning/README.md | 18 ++--- 5-app-infra/README.md | 2 +- .../projects/artifact-publish/README.md | 20 +++--- .../projects/machine-learning/README.md | 18 ++--- .../projects/service-catalog/README.md | 22 +++--- ERRATA.md | 2 +- README.md | 4 +- docs/FAQ.md | 2 +- docs/TROUBLESHOOTING.md | 4 +- docs/upgrading_to_v2.0.md | 2 +- docs/upgrading_to_v3.0.md | 4 +- helpers/foundation-deployer/README.md | 26 +++---- helpers/foundation-deployer/gcp/gcp.go | 2 +- .../foundation-deployer/global.tfvars.example | 18 ++--- helpers/foundation-deployer/go.mod | 4 +- helpers/foundation-deployer/go.sum | 4 +- helpers/foundation-deployer/main.go | 10 +-- helpers/foundation-deployer/msg/msg.go | 4 +- helpers/foundation-deployer/stages/apply.go | 10 +-- helpers/foundation-deployer/stages/data.go | 2 +- helpers/foundation-deployer/stages/destroy.go | 12 ++-- .../foundation-deployer/stages/validate.go | 2 +- scripts/set-tfc-backend-and-remote.sh | 2 +- scripts/validate-requirements.sh | 4 +- test/integration/bootstrap/bootstrap_test.go | 4 +- test/integration/envs/envs_test.go | 2 +- test/integration/go.mod | 2 +- test/integration/networks/networks_test.go | 2 +- test/integration/org/org_test.go | 2 +- .../projects-shared/projects_shared_test.go | 2 +- test/integration/projects/projects_test.go | 2 +- test/integration/shared/shared_test.go | 2 +- 48 files changed, 268 insertions(+), 268 deletions(-) diff --git a/0-bootstrap/README-GitHub.md b/0-bootstrap/README-GitHub.md index 27737b9b..f7e635e4 100644 --- a/0-bootstrap/README-GitHub.md +++ b/0-bootstrap/README-GitHub.md @@ -61,13 +61,13 @@ for each one of the repositories. ### Deploying step 0-bootstrap -1. Clone [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation) into your local environment. +1. Clone [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) into your local environment. ```bash - git clone https://github.com/terraform-google-modules/terraform-example-foundation.git + git clone https://github.com/terraform-google-modules/terraform-google-enterprise-genai.git ``` -1. Clone the private repository you created to host the `0-bootstrap` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the private repository you created to host the `0-bootstrap` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. You must be [authenticated to GitHub](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-authentication-to-github). ```bash @@ -78,7 +78,7 @@ You must be [authenticated to GitHub](https://docs.github.com/en/authentication/ ```bash gcp-bootstrap/ - terraform-example-foundation/ + terraform-google-enterprise-genai/ ``` 1. Navigate into the repo. All subsequent @@ -110,11 +110,11 @@ You must be [authenticated to GitHub](https://docs.github.com/en/authentication/ ```bash mkdir -p envs/shared - cp -RT ../terraform-example-foundation/0-bootstrap/ ./envs/shared - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/0-bootstrap/ ./envs/shared + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh cd ./envs/shared ``` @@ -153,7 +153,7 @@ export the GitHub fine grained access token as an environment variable: 1. Use the helper script [validate-requirements.sh](../scripts/validate-requirements.sh) to validate your environment: ```bash - ../../../terraform-example-foundation/scripts/validate-requirements.sh -o -b -u -e + ../../../terraform-google-enterprise-genai/scripts/validate-requirements.sh -o -b -u -e ``` **Note:** The script is not able to validate if the user is in a Cloud Identity or Google Workspace group with the required roles. @@ -249,7 +249,7 @@ we recommend that you request 50 additional projects for the **projects step ser ## Deploying step 1-org -1. Clone the repository you created to host the `1-org` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the repository you created to host the `1-org` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. ```bash git clone git@github.com:/.git gcp-org @@ -281,11 +281,11 @@ we recommend that you request 50 additional projects for the **projects step ser 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/1-org/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/1-org/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -365,7 +365,7 @@ See the shared folder [README.md](../1-org/envs/shared/README.md#inputs) for add ## Deploying step 2-environments -1. Clone the repository you created to host the `2-environments` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the repository you created to host the `2-environments` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. ```bash git clone git@github.com:/.git gcp-environments @@ -404,11 +404,11 @@ See the shared folder [README.md](../1-org/envs/shared/README.md#inputs) for add 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/2-environments/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/2-environments/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -472,7 +472,7 @@ or go to [Deploying step 3-networks-hub-and-spoke](#deploying-step-3-networks-hu ## Deploying step 3-networks-dual-svpc -1. Clone the repository you created to host the `3-networks-dual-svpc` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the repository you created to host the `3-networks-dual-svpc` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. ```bash git clone git@github.com:/.git gcp-networks @@ -510,11 +510,11 @@ or go to [Deploying step 3-networks-hub-and-spoke](#deploying-step-3-networks-hu 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/3-networks-dual-svpc/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -635,7 +635,7 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th ## Deploying step 3-networks-hub-and-spoke -1. Clone the repository you created to host the `3-networks-hub-and-spoke` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the repository you created to host the `3-networks-hub-and-spoke` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. ```bash git clone git@github.com:/.git gcp-networks @@ -673,11 +673,11 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-hub-and-spoke/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/3-networks-hub-and-spoke/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -786,7 +786,7 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th ## Deploying step 4-projects -1. Clone the repository you created to host the `4-projects` terraform configuration at the same level of the `terraform-example-foundation` folder. +1. Clone the repository you created to host the `4-projects` terraform configuration at the same level of the `terraform-google-enterprise-genai` folder. ```bash git clone git@github.com:/.git gcp-projects @@ -825,11 +825,11 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/4-projects/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library + cp -RT ../terraform-google-enterprise-genai/4-projects/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library mkdir -p .github/workflows - cp ../terraform-example-foundation/build/github-tf-* ./.github/workflows/ - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp ../terraform-google-enterprise-genai/build/github-tf-* ./.github/workflows/ + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/0-bootstrap/README-Jenkins.md b/0-bootstrap/README-Jenkins.md index 497e7137..77e081ef 100644 --- a/0-bootstrap/README-Jenkins.md +++ b/0-bootstrap/README-Jenkins.md @@ -51,7 +51,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap` - Access to the Jenkins Controller Web UI - [SSH Agent Jenkins plugin](https://plugins.jenkins.io/ssh-agent) installed in your Jenkins Controller - Private IP address for the Jenkins Agent: usually assigned by your network administrator. You will use this IP for the GCE instance that will be created in the `prj-b-cicd` GCP Project in step [II. Create the SEED and CI/CD projects using Terraform](#ii-create-the-seed-and-cicd-projects-using-terraform). - - Access to create five Git repositories, one for each directory in this [monorepo](https://github.com/terraform-google-modules/terraform-example-foundation) (`0-bootstrap, 1-org, 2-environments, 3-networks, 4-projects`). These are usually private repositories that might be on-prem. + - Access to create five Git repositories, one for each directory in this [monorepo](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) (`0-bootstrap, 1-org, 2-environments, 3-networks, 4-projects`). These are usually private repositories that might be on-prem. 1. Generate a SSH key pair. In the Jenkins Controller host, use the `ssh-keygen` command to generate a SSH key pair. - You will need this key pair to enable authentication between the Controller and Agent. Although the key pair can be generated in any linux machine, it is recommended not to copy the secret private key from one host to another, so you probably want to do this in the Jenkins Controller host command line. @@ -78,7 +78,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap` - Jenkins Agent’s private IP address (usually assigned by your Network Administrator. In the provided examples this IP is "172.16.1.6"). This private IP will be reachable through the VPN connection that you will create later. 1. Create five individual Git repositories in your Git server (This might be a task delegated to your infrastructure team) - - Note that although this infrastructure code is distributed to you as a [monorepo](https://github.com/terraform-google-modules/terraform-example-foundation), you will store the code in five different repositories, one for each directory: + - Note that although this infrastructure code is distributed to you as a [monorepo](https://github.com/terraform-google-modules/terraform-google-enterprise-genai), you will store the code in five different repositories, one for each directory: ```text ./0-bootstrap @@ -113,7 +113,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap` 1. Clone this mono-repository with: ```bash - git clone https://github.com/terraform-google-modules/terraform-example-foundation + git clone https://github.com/terraform-google-modules/terraform-google-enterprise-genai ``` 1. Clone the repository you created to host the `0-bootstrap` directory with: @@ -132,7 +132,7 @@ You arrived to these instructions because you are using the `jenkins_bootstrap` 1. Copy contents of foundation to new repo (modify accordingly based on your current directory). ```bash - cp -RT ../terraform-example-foundation/0-bootstrap/ . + cp -RT ../terraform-google-enterprise-genai/0-bootstrap/ . ``` 1. Activate the Jenkins module and disable the Cloud Build module. This implies manually editing the following files: @@ -313,10 +313,10 @@ Here you will configure a VPN Network tunnel to enable connectivity between the 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/1-org/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/Jenkinsfile . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/1-org/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/Jenkinsfile . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -419,10 +419,10 @@ Here you will configure a VPN Network tunnel to enable connectivity between the 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/2-environments/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/Jenkinsfile . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/2-environments/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/Jenkinsfile . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -525,10 +525,10 @@ Here you will configure a VPN Network tunnel to enable connectivity between the 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/Jenkinsfile . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/3-networks-dual-svpc/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/Jenkinsfile . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -678,10 +678,10 @@ Here you will configure a VPN Network tunnel to enable connectivity between the 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-hub-and-spoke/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/Jenkinsfile . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/3-networks-hub-and-spoke/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/Jenkinsfile . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -831,10 +831,10 @@ Here you will configure a VPN Network tunnel to enable connectivity between the 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/4-projects/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/Jenkinsfile . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/4-projects/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/Jenkinsfile . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/0-bootstrap/README-Terraform-Cloud.md b/0-bootstrap/README-Terraform-Cloud.md index ce5794ec..935f3e78 100644 --- a/0-bootstrap/README-Terraform-Cloud.md +++ b/0-bootstrap/README-Terraform-Cloud.md @@ -59,13 +59,13 @@ that are created, see the organization bootstrap module ### Instructions -1. Clone [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation) into your local environment. +1. Clone [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) into your local environment. ```bash - git clone https://github.com/terraform-google-modules/terraform-example-foundation.git + git clone https://github.com/terraform-google-modules/terraform-google-enterprise-genai.git ``` -1. Clone all the private repositories (or projects) you created at the same level of the `terraform-example-foundation` folder. +1. Clone all the private repositories (or projects) you created at the same level of the `terraform-google-enterprise-genai` folder. You must be authenticated to the VCS provider. See [GitHub authentication](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-authentication-to-github) or [GitLab authentication](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-authentication-to-github) for more details. ```bash @@ -92,7 +92,7 @@ You must be authenticated to the VCS provider. See [GitHub authentication](https gcp-environments/ gcp-networks/ gcp-projects/ - terraform-example-foundation/ + terraform-google-enterprise-genai/ ``` 1. In your VCS repositories (or projects) it is expected to have the following branches created. Also, these branches shouldn't be empty, you need at least a single file. Run `scripts/git_create_branches_helper.sh` script to create these branches with a seed file for each repository automatically. @@ -104,8 +104,8 @@ You must be authenticated to the VCS provider. See [GitHub authentication](https - Note: `scripts/git_create_branches_helper.sh` script and the following commands assume you are running it from the directory that has all the repos cloned (layout described in the previous step). If you run from another directory, adjust the `BASE_PATH` variable at the `scripts/git_create_branches_helper.sh` and adjust in the following commands. ```bash - chmod 755 ./terraform-example-foundation/0-bootstrap/scripts/git_create_branches_helper.sh - ./terraform-example-foundation/0-bootstrap/scripts/git_create_branches_helper.sh + chmod 755 ./terraform-google-enterprise-genai/0-bootstrap/scripts/git_create_branches_helper.sh + ./terraform-google-enterprise-genai/0-bootstrap/scripts/git_create_branches_helper.sh ``` You will see some GIT logs related to the branches creation in the console and the message `"Branch creation and push completed for all repositories"` at the end of the script execution. @@ -130,7 +130,7 @@ You must be authenticated to the VCS provider. See [GitHub authentication](https ```bash mkdir -p envs/shared - cp -RT ../terraform-example-foundation/0-bootstrap/ ./envs/shared + cp -RT ../terraform-google-enterprise-genai/0-bootstrap/ ./envs/shared cd ./envs/shared ``` @@ -187,7 +187,7 @@ export the OAuth Token ID as an environment variable: 1. Use the helper script [validate-requirements.sh](../scripts/validate-requirements.sh) to validate your environment: ```bash - ../../../terraform-example-foundation/scripts/validate-requirements.sh -o -b -u -e + ../../../terraform-google-enterprise-genai/scripts/validate-requirements.sh -o -b -u -e ``` **Note:** The script is not able to validate if the user is in a Cloud Identity or Google Workspace group with the required roles. @@ -244,8 +244,8 @@ export the OAuth Token ID as an environment variable: ```bash mv backend.tf.cloud.example backend.tf cd ../../../ - chmod 775 ./terraform-example-foundation/scripts/set-tfc-backend-and-remote.sh - ./terraform-example-foundation/scripts/set-tfc-backend-and-remote.sh + chmod 775 ./terraform-google-enterprise-genai/scripts/set-tfc-backend-and-remote.sh + ./terraform-google-enterprise-genai/scripts/set-tfc-backend-and-remote.sh cd ./gcp-bootstrap/envs/shared ``` @@ -293,7 +293,7 @@ we recommend that you request 50 additional projects for the **projects step ser 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/1-org/ . + cp -RT ../terraform-google-enterprise-genai/1-org/ . ``` 1. Rename `./envs/shared/terraform.example.tfvars` to `./envs/shared/terraform.tfvars` @@ -379,7 +379,7 @@ See the shared folder [README.md](../1-org/envs/shared/README.md#inputs) for add 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/2-environments/ . + cp -RT ../terraform-google-enterprise-genai/2-environments/ . ``` 1. Rename `terraform.example.tfvars` to `terraform.tfvars`. @@ -452,9 +452,9 @@ or go to [Deploying step 3-networks-hub-and-spoke](#deploying-step-3-networks-hu 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/3-networks-dual-svpc/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -609,9 +609,9 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/3-networks-hub-and-spoke/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/3-networks-hub-and-spoke/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -755,9 +755,9 @@ An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set with th 1. Copy contents of foundation to new repo. ```bash - cp -RT ../terraform-example-foundation/4-projects/ . - cp -RT ../terraform-example-foundation/policy-library/ ./policy-library - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/4-projects/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ ./policy-library + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/0-bootstrap/README.md b/0-bootstrap/README.md index 721c2998..7048283f 100644 --- a/0-bootstrap/README.md +++ b/0-bootstrap/README.md @@ -51,7 +51,7 @@ Hub and Spoke network model. It also sets up the global DNS hub. For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -119,7 +119,7 @@ See [troubleshooting](../docs/TROUBLESHOOTING.md) if you run into issues during ## Deploying with Jenkins If you are using the `jenkins_bootstrap` sub-module, see -[README-Jenkins](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README-Jenkins.md) +[README-Jenkins](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README-Jenkins.md) for requirements and instructions on how to run the 0-bootstrap step. Using Jenkins requires a few manual steps, including configuring connectivity with your current Jenkins manager (controller) environment. @@ -132,12 +132,12 @@ Using GitHub Actions requires manual creation of the GitHub repositories used in ## Deploying with Cloud Build -1. Clone [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation) into your local environment and navigate to the `0-bootstrap` folder. +1. Clone [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) into your local environment and navigate to the `0-bootstrap` folder. ```bash - git clone https://github.com/terraform-google-modules/terraform-example-foundation.git + git clone https://github.com/terraform-google-modules/terraform-google-enterprise-genai.git - cd terraform-example-foundation/0-bootstrap + cd terraform-google-enterprise-genai/0-bootstrap ``` 1. Rename `terraform.example.tfvars` to `terraform.tfvars` and update the file with values from your environment: @@ -223,7 +223,7 @@ Using GitHub Actions requires manual creation of the GitHub repositories used in ``` 1. (Optional) Run `terraform plan` to verify that state is configured correctly. You should see no changes from the previous state. -1. Clone the policy repo and copy contents of policy-library to new repo. Clone the repo at the same level of the `terraform-example-foundation` folder. +1. Clone the policy repo and copy contents of policy-library to new repo. Clone the repo at the same level of the `terraform-google-enterprise-genai` folder. ```bash cd ../.. @@ -232,7 +232,7 @@ Using GitHub Actions requires manual creation of the GitHub repositories used in cd gcp-policies git checkout -b main - cp -RT ../terraform-example-foundation/policy-library/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ . ``` 1. Commit changes and push your main branch to the policy repo. @@ -258,9 +258,9 @@ Using GitHub Actions requires manual creation of the GitHub repositories used in git checkout -b plan mkdir -p envs/shared - cp -RT ../terraform-example-foundation/0-bootstrap/ ./envs/shared - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/0-bootstrap/ ./envs/shared + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh git add . diff --git a/1-org/README.md b/1-org/README.md index d5e28640..d528b9db 100644 --- a/1-org/README.md +++ b/1-org/README.md @@ -51,7 +51,7 @@ hub-and-spoke network model. It also sets up the global DNS hub. For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation). +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai). ## Purpose @@ -113,11 +113,11 @@ This module creates and applies [tags](https://cloud.google.com/resource-manager ### Deploying with Cloud Build 1. Clone the `gcp-org` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder. If required, run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} gcloud source repos clone gcp-org --project=${CLOUD_BUILD_PROJECT_ID} @@ -134,9 +134,9 @@ If required, run `terraform output cloudbuild_project_id` in the `0-bootstrap` f cd gcp-org git checkout -b plan - cp -RT ../terraform-example-foundation/1-org/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/1-org/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -149,7 +149,7 @@ If required, run `terraform output cloudbuild_project_id` in the `0-bootstrap` f 1. Check if a Security Command Center notification with the default name, **scc-notify**, already exists. If it exists, choose a different value for the `scc_notification_name` variable in the `./envs/shared/terraform.tfvars` file. ```bash - export ORGANIZATION_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) + export ORGANIZATION_ID=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) gcloud scc notifications describe "scc-notify" --organization=${ORGANIZATION_ID} ``` @@ -163,7 +163,7 @@ If required, run `terraform output cloudbuild_project_id` in the `0-bootstrap` f 1. Update the `envs/shared/terraform.tfvars` file with values from your environment and 0-bootstrap step. If the previous step showed a numeric value, un-comment the variable `create_access_context_manager_access_policy = false`. See the shared folder [README.md](./envs/shared/README.md) for additional information on the values in the `terraform.tfvars` file. ```bash - export backend_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate) + export backend_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw gcs_bucket_tfstate) echo "remote_state_bucket = ${backend_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./envs/shared/terraform.tfvars @@ -200,7 +200,7 @@ If required, run `terraform output cloudbuild_project_id` in the `0-bootstrap` f If you received a `PERMISSION_DENIED` error while running the `gcloud access-context-manager` or the `gcloud scc notifications` commands, you can append the following to run the command as the Terraform service account: ```bash ---impersonate-service-account=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw organization_step_terraform_service_account_email) +--impersonate-service-account=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw organization_step_terraform_service_account_email) ``` ### Deploying with Jenkins @@ -213,11 +213,11 @@ See `0-bootstrap` [README-GitHub.md](../0-bootstrap/README-GitHub.md#deploying-s ### Running Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into the `1-org` folder, copy the Terraform wrapper script, and ensure it can be executed. ```bash - cd terraform-example-foundation/1-org + cd terraform-google-enterprise-genai/1-org cp ../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/1-org/modules/cai-monitoring/README.md b/1-org/modules/cai-monitoring/README.md index 98e53a95..1e1f0344 100644 --- a/1-org/modules/cai-monitoring/README.md +++ b/1-org/modules/cai-monitoring/README.md @@ -5,7 +5,7 @@ Uses Google Cloud Asset Inventory to create a feed of IAM Policy change events, ```hcl module "secure_cai_notification" { - source = "terraform-google-modules/terraform-example-foundation/google//1-org/modules/cai-monitoring" + source = "terraform-google-modules/terraform-google-enterprise-genai/google//1-org/modules/cai-monitoring" org_id = billing_account = diff --git a/1-org/modules/centralized-logging/README.md b/1-org/modules/centralized-logging/README.md index e6fab9bc..d49cf9bb 100644 --- a/1-org/modules/centralized-logging/README.md +++ b/1-org/modules/centralized-logging/README.md @@ -10,7 +10,7 @@ The following example exports audit logs from two folders to the same storage de ```hcl module "logs_export" { - source = "terraform-google-modules/terraform-example-foundation/google//1-org/modules/centralized-logging" + source = "terraform-google-modules/terraform-google-enterprise-genai/google//1-org/modules/centralized-logging" resources = { fldr1 = "" @@ -35,7 +35,7 @@ The following example exports all logs from three projects - including the loggi ```hcl module "logging_logbucket" { - source = "terraform-google-modules/terraform-example-foundation/google//1-org/modules/centralized-logging" + source = "terraform-google-modules/terraform-google-enterprise-genai/google//1-org/modules/centralized-logging" resources = { prj1 = "" diff --git a/2-environments/README.md b/2-environments/README.md index 352088c5..ccabb5fd 100644 --- a/2-environments/README.md +++ b/2-environments/README.md @@ -51,7 +51,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation). +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai). ## Purpose @@ -90,11 +90,11 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Deploying with Cloud Build 1. Clone the `gcp-environments` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} gcloud source repos clone gcp-environments --project=${CLOUD_BUILD_PROJECT_ID} @@ -108,9 +108,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-environments git checkout -b plan - cp -RT ../terraform-example-foundation/2-environments/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/2-environments/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -123,7 +123,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Update the file with values from your environment and bootstrap (you can re-run `terraform output` in the 0-bootstrap directory to find these values). See any of the envs folder [README.md](./envs/production/README.md#inputs) files for additional information on the values in the `terraform.tfvars` file. ```bash - export backend_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate) + export backend_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw gcs_bucket_tfstate) echo "remote_state_bucket = ${backend_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${backend_bucket}/" terraform.tfvars @@ -212,10 +212,10 @@ See `0-bootstrap` [README-GitHub.md](../0-bootstrap/README-GitHub.md#deploying-s ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `2-environments` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `2-environments` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/2-environments + cd terraform-google-enterprise-genai/2-environments cp ../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/3-networks-dual-svpc/README.md b/3-networks-dual-svpc/README.md index 3adb5836..cb47dcf7 100644 --- a/3-networks-dual-svpc/README.md +++ b/3-networks-dual-svpc/README.md @@ -51,7 +51,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation). +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai). ## Purpose @@ -65,10 +65,10 @@ The purpose of this step is to: 1. 0-bootstrap executed successfully. 1. 1-org executed successfully. 1. 2-environments executed successfully. -1. Obtain the value for the access_context_manager_policy_id variable. It can be obtained by running the following commands. We assume you are at the same level as directory `terraform-example-foundation`, If you run them from another directory, adjust your paths accordingly. +1. Obtain the value for the access_context_manager_policy_id variable. It can be obtained by running the following commands. We assume you are at the same level as directory `terraform-google-enterprise-genai`, If you run them from another directory, adjust your paths accordingly. ```bash - export ORGANIZATION_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) + export ORGANIZATION_ID=$(terraform -chdir="terraform-google-enterprise-genai/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)") echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}" ``` @@ -142,11 +142,11 @@ If you are not able to use Dedicated or Partner Interconnect, you can also use a ### Deploying with Cloud Build 1. Clone the `gcp-networks` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} gcloud source repos clone gcp-networks --project=${CLOUD_BUILD_PROJECT_ID} @@ -158,9 +158,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-networks/ git checkout -b plan - cp -RT ../terraform-example-foundation/3-networks-dual-svpc/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/3-networks-dual-svpc/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -178,13 +178,13 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get Use `terraform output` to get the backend bucket value from 0-bootstrap output. ```bash - export ORGANIZATION_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) + export ORGANIZATION_ID=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -json common_config | jq '.org_id' --raw-output) export ACCESS_CONTEXT_MANAGER_ID=$(gcloud access-context-manager policies list --organization ${ORGANIZATION_ID} --format="value(name)") echo "access_context_manager_policy_id = ${ACCESS_CONTEXT_MANAGER_ID}" sed -i "s/ACCESS_CONTEXT_MANAGER_ID/${ACCESS_CONTEXT_MANAGER_ID}/" ./access_context.auto.tfvars - export backend_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate) + export backend_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw gcs_bucket_tfstate) echo "remote_state_bucket = ${backend_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${backend_bucket}/" ./common.auto.tfvars @@ -203,10 +203,10 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Use `terraform output` to get the Cloud Build project ID and the networks step Terraform Service Account from 0-bootstrap output. An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set using the Terraform Service Account to enable impersonation. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} - export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw networks_step_terraform_service_account_email) + export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw networks_step_terraform_service_account_email) echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT} ``` @@ -286,10 +286,10 @@ See `0-bootstrap` [README-GitHub.md](../0-bootstrap/README-GitHub.md#deploying-s ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `3-networks-dual-svpc` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `3-networks-dual-svpc` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/3-networks-dual-svpc + cd terraform-google-enterprise-genai/3-networks-dual-svpc cp ../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/4-projects/README.md b/4-projects/README.md index 2af08403..f5ff9c88 100644 --- a/4-projects/README.md +++ b/4-projects/README.md @@ -51,7 +51,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation). +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai). ## Purpose @@ -59,8 +59,8 @@ The purpose of this step is to set up the folder structure, projects, and infras For each business unit, a shared `infra-pipeline` project is created along with Cloud Build triggers, CSRs for application infrastructure code and Google Cloud Storage buckets for state storage. -This step follows the same [conventions](https://github.com/terraform-google-modules/terraform-example-foundation#branching-strategy) as the Foundation pipeline deployed in [0-bootstrap](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md). -A custom [workspace](https://github.com/terraform-google-modules/terraform-google-bootstrap/blob/master/modules/tf_cloudbuild_workspace/README.md) (`bu1-example-app`) is created by this pipeline and necessary roles are granted to the Terraform Service Account of this workspace by enabling variable `sa_roles` as shown in this [example](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects/modules/base_env/example_base_shared_vpc_project.tf). +This step follows the same [conventions](https://github.com/terraform-google-modules/terraform-google-enterprise-genai#branching-strategy) as the Foundation pipeline deployed in [0-bootstrap](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README.md). +A custom [workspace](https://github.com/terraform-google-modules/terraform-google-bootstrap/blob/master/modules/tf_cloudbuild_workspace/README.md) (`bu1-example-app`) is created by this pipeline and necessary roles are granted to the Terraform Service Account of this workspace by enabling variable `sa_roles` as shown in this [example](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/4-projects/modules/base_env/example_base_shared_vpc_project.tf). This pipeline is utilized to deploy resources in projects across development/non-production/production in step [5-app-infra](../5-app-infra/README.md). Other Workspaces can also be created to isolate deployments if needed. @@ -90,11 +90,11 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Deploying with Cloud Build 1. Clone the `gcp-projects` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} gcloud source repos clone gcp-projects --project=${CLOUD_BUILD_PROJECT_ID} @@ -106,9 +106,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-projects git checkout -b plan - cp -RT ../terraform-example-foundation/4-projects/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/4-projects/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -128,7 +128,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Use `terraform output` to get the backend bucket value from 0-bootstrap output. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars @@ -146,10 +146,10 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Use `terraform output` to get the Cloud Build project ID and the projects step Terraform Service Account from 0-bootstrap output. An environment variable `GOOGLE_IMPERSONATE_SERVICE_ACCOUNT` will be set using the Terraform Service Account to enable impersonation. ```bash - export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw cloudbuild_project_id) + export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw cloudbuild_project_id) echo ${CLOUD_BUILD_PROJECT_ID} - export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_step_terraform_service_account_email) + export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_step_terraform_service_account_email) echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT} ``` @@ -235,10 +235,10 @@ See `0-bootstrap` [README-GitHub.md](../0-bootstrap/README-GitHub.md#deploying-s ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `4-projects` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `4-projects` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/4-projects + cd terraform-google-enterprise-genai/4-projects cp ../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/0-gcp-policies/README.md b/5-app-infra/0-gcp-policies/README.md index 3995e7c9..8e75a01e 100644 --- a/5-app-infra/0-gcp-policies/README.md +++ b/5-app-infra/0-gcp-policies/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -62,7 +62,7 @@ When using Cloud Build for deployment, Policy Verification must be checked. Thi 1. Ensure you are in a neutral directory outside any other git related repositories. 1. Clone the `gcp-policies` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash diff --git a/5-app-infra/1-artifact-publish/README.md b/5-app-infra/1-artifact-publish/README.md index 6d592a31..51f0be86 100644 --- a/5-app-infra/1-artifact-publish/README.md +++ b/5-app-infra/1-artifact-publish/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -76,7 +76,7 @@ each folder under `images` has the full name and tag of the image that must be b Once pushed, the pipeline can be accessed by navigating to the project name created in step-4: ```bash -terraform -chdir="../terraform-example-foundation/4-projects/business_unit_3/shared/" output -raw common_artifacts_project_id +terraform -chdir="../terraform-google-enterprise-genai/4-projects/business_unit_3/shared/" output -raw common_artifacts_project_id ``` ## Prerequisites @@ -112,9 +112,9 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. cd bu3-artifact-publish git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/1-artifact-publish/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/1-artifact-publish/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -127,7 +127,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. 1. Update the file with values from your environment and 0-bootstrap. See any of the business unit 1 envs folders [README.md](./business_unit_1/production/README.md) files for additional information on the values in the `common.auto.tfvars` file. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -169,10 +169,10 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/artifact-publish + cd terraform-google-enterprise-genai/5-app-infra/projects/artifact-publish cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/2-artifact-publish-repo/README.md b/5-app-infra/2-artifact-publish-repo/README.md index c08fc84f..9cdaea98 100644 --- a/5-app-infra/2-artifact-publish-repo/README.md +++ b/5-app-infra/2-artifact-publish-repo/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Usage diff --git a/5-app-infra/3-service-catalog/README.md b/5-app-infra/3-service-catalog/README.md index 8333b28a..f52174d2 100644 --- a/5-app-infra/3-service-catalog/README.md +++ b/5-app-infra/3-service-catalog/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose(s) @@ -111,7 +111,7 @@ The pipeline also listens for changes made to `plan`, `development`, `non-produc The pipeline can be accessed by navigating to the project name created in step-4: ```bash -terraform -chdir="../terraform-example-foundation/4-projects/business_unit_3/shared/" output -raw service_catalog_project_id +terraform -chdir="../terraform-google-enterprise-genai/4-projects/business_unit_3/shared/" output -raw service_catalog_project_id ``` ## Prerequisites @@ -146,9 +146,9 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. cd bu3-service-catalog git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/3-service-catalog/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/3-service-catalog/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -161,7 +161,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. 1. Update the file with values from your environment and 0-bootstrap. See any of the business unit 1 envs folders [README.md](./business_unit_1/production/README.md) files for additional information on the values in the `common.auto.tfvars` file. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -205,10 +205,10 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/service-catalog + cd terraform-google-enterprise-genai/5-app-infra/projects/service-catalog cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/4-service-catalog-repo/README.md b/5-app-infra/4-service-catalog-repo/README.md index ebdacf3d..a7e93409 100644 --- a/5-app-infra/4-service-catalog-repo/README.md +++ b/5-app-infra/4-service-catalog-repo/README.md @@ -60,7 +60,7 @@ This repo provides a number of the [Google Service Catalog](https://cloud.google 1. Enter the repo folder and copy over the service catalogs files from `5-app-infra/4-service-catalog-repo` folder. ```shell cd service-catalog - cp -RT ../terraform-example-foundation/5-app-infra/4-service-catalog-repo/ . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/4-service-catalog-repo/ . ``` 1. Commit changes and push main branch to the new repo. diff --git a/5-app-infra/5-vpc-sc/README.md b/5-app-infra/5-vpc-sc/README.md index bb13c634..88fd2d0d 100644 --- a/5-app-infra/5-vpc-sc/README.md +++ b/5-app-infra/5-vpc-sc/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## VPC-SC diff --git a/5-app-infra/6-machine-learning/README.md b/5-app-infra/6-machine-learning/README.md index ce5c33ff..e10dd1e9 100644 --- a/5-app-infra/6-machine-learning/README.md +++ b/5-app-infra/6-machine-learning/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -95,7 +95,7 @@ Step 12 in "Deploying with Cloud Build" highlights the necessary steps needed to ### Deploying with Cloud Build 1. Clone the `gcp-policies` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash @@ -115,7 +115,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-policies-app-infra git checkout -b main - cp -RT ../terraform-example-foundation/policy-library/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ . ``` 1. Commit changes and push your main branch to the new repo. @@ -147,9 +147,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd bu3-machine-learning git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/6-machine-learning/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/6-machine-learning/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -171,7 +171,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Use `terraform output` to get the project backend bucket value from 0-bootstrap. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -268,10 +268,10 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get ``` ## Running Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/machine-learning + cd terraform-google-enterprise-genai/5-app-infra/projects/machine-learning cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/README.md b/5-app-infra/README.md index 4fdc1e94..b41e5da6 100644 --- a/5-app-infra/README.md +++ b/5-app-infra/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose diff --git a/5-app-infra/projects/artifact-publish/README.md b/5-app-infra/projects/artifact-publish/README.md index 7e44405d..eb6ed8bc 100644 --- a/5-app-infra/projects/artifact-publish/README.md +++ b/5-app-infra/projects/artifact-publish/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -76,7 +76,7 @@ each folder under `images` has the full name and tag of the image that must be b Once pushed, the pipeline can be accessed by navigating to the project name created in step-4: ```bash -terraform -chdir="../terraform-example-foundation/4-projects/business_unit_3/shared/" output -raw common_artifacts_project_id +terraform -chdir="../terraform-google-enterprise-genai/4-projects/business_unit_3/shared/" output -raw common_artifacts_project_id ``` ## Prerequisites @@ -99,7 +99,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Deploying with Cloud Build 1. Clone the `gcp-policies` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash @@ -119,7 +119,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-policies-app-infra git checkout -b main - cp -RT ../terraform-example-foundation/policy-library/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ . ``` 1. Commit changes and push your main branch to the new repo. @@ -151,9 +151,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd bu3-artifact-publish git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/projects/artifact-publish/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/projects/artifact-publish/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -166,7 +166,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Update the file with values from your environment and 0-bootstrap. See any of the business unit 1 envs folders [README.md](./business_unit_1/production/README.md) files for additional information on the values in the `common.auto.tfvars` file. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -233,10 +233,10 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/artifact-publish + cd terraform-google-enterprise-genai/5-app-infra/projects/artifact-publish cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/projects/machine-learning/README.md b/5-app-infra/projects/machine-learning/README.md index 6475afac..91bb1602 100644 --- a/5-app-infra/projects/machine-learning/README.md +++ b/5-app-infra/projects/machine-learning/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose @@ -96,7 +96,7 @@ Step 12 in "Deploying with Cloud Build" highlights the necessary steps needed to ### Deploying with Cloud Build 1. Clone the `gcp-policies` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash @@ -116,7 +116,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-policies-app-infra git checkout -b main - cp -RT ../terraform-example-foundation/policy-library/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ . ``` 1. Commit changes and push your main branch to the new repo. @@ -148,9 +148,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd bu3-machine-learning git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/projects/machine-learning/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/projects/machine-learning/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -165,7 +165,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Use `terraform output` to get the project backend bucket value from 0-bootstrap. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -315,10 +315,10 @@ we want the `unknown-project-number` here. Add this into your `egress_policies` ## Running Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/machine-learning + cd terraform-google-enterprise-genai/5-app-infra/projects/machine-learning cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/5-app-infra/projects/service-catalog/README.md b/5-app-infra/projects/service-catalog/README.md index a309c3f0..4a70e7df 100644 --- a/5-app-infra/projects/service-catalog/README.md +++ b/5-app-infra/projects/service-catalog/README.md @@ -50,7 +50,7 @@ Hub and Spoke network model. It also sets up the global DNS hub For an overview of the architecture and the parts, see the -[terraform-example-foundation README](https://github.com/terraform-google-modules/terraform-example-foundation) +[terraform-google-enterprise-genai README](https://github.com/terraform-google-modules/terraform-google-enterprise-genai) file. ## Purpose(s) @@ -111,7 +111,7 @@ The pipeline also listens for changes made to `plan`, `development`, `non-produc The pipeline can be accessed by navigating to the project name created in step-4: ```bash -terraform -chdir="../terraform-example-foundation/4-projects/business_unit_3/shared/" output -raw service_catalog_project_id +terraform -chdir="../terraform-google-enterprise-genai/4-projects/business_unit_3/shared/" output -raw service_catalog_project_id ``` ## Prerequisites @@ -133,7 +133,7 @@ commands. The `-T` flag is needed for Linux, but causes problems for MacOS. ### Deploying with Cloud Build 1. Clone the `gcp-policies` repo based on the Terraform output from the `0-bootstrap` step. -Clone the repo at the same level of the `terraform-example-foundation` folder, the following instructions assume this layout. +Clone the repo at the same level of the `terraform-google-enterprise-genai` folder, the following instructions assume this layout. Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get the Cloud Build Project ID. ```bash @@ -153,7 +153,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd gcp-policies-app-infra git checkout -b main - cp -RT ../terraform-example-foundation/policy-library/ . + cp -RT ../terraform-google-enterprise-genai/policy-library/ . ``` 1. Commit changes and push your main branch to the new repo. @@ -185,9 +185,9 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get cd bu3-service-catalog git checkout -b plan - cp -RT ../terraform-example-foundation/5-app-infra/projects/service-catalog/ . - cp ../terraform-example-foundation/build/cloudbuild-tf-* . - cp ../terraform-example-foundation/build/tf-wrapper.sh . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/projects/service-catalog/ . + cp ../terraform-google-enterprise-genai/build/cloudbuild-tf-* . + cp ../terraform-google-enterprise-genai/build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` @@ -200,7 +200,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Update the file with values from your environment and 0-bootstrap. See any of the business unit 1 envs folders [README.md](./business_unit_1/production/README.md) files for additional information on the values in the `common.auto.tfvars` file. ```bash - export remote_state_bucket=$(terraform -chdir="../terraform-example-foundation/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) + export remote_state_bucket=$(terraform -chdir="../terraform-google-enterprise-genai/0-bootstrap/" output -raw projects_gcs_bucket_tfstate) echo "remote_state_bucket = ${remote_state_bucket}" sed -i "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars ``` @@ -253,7 +253,7 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get 1. Enter the repo folder and copy over the service catalogs files from `5-app-infra/source_repos` folder. ```shell cd service-catalog - cp -RT ../terraform-example-foundation/5-app-infra/source_repos/service-catalog/ . + cp -RT ../terraform-google-enterprise-genai/5-app-infra/source_repos/service-catalog/ . ``` 1. Commit changes and push main branch to the new repo. @@ -268,10 +268,10 @@ Run `terraform output cloudbuild_project_id` in the `0-bootstrap` folder to get ### Run Terraform locally -1. The next instructions assume that you are at the same level of the `terraform-example-foundation` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. +1. The next instructions assume that you are at the same level of the `terraform-google-enterprise-genai` folder. Change into `5-app-infra` folder, copy the Terraform wrapper script and ensure it can be executed. ```bash - cd terraform-example-foundation/5-app-infra/projects/service-catalog + cd terraform-google-enterprise-genai/5-app-infra/projects/service-catalog cp ../../../build/tf-wrapper.sh . chmod 755 ./tf-wrapper.sh ``` diff --git a/ERRATA.md b/ERRATA.md index 1f44ea25..a00986a7 100644 --- a/ERRATA.md +++ b/ERRATA.md @@ -35,7 +35,7 @@ This will be addressed in the next version of the blueprint guide. - Cloud Asset Inventory will be integrated in a future release. - The unallocated IP address space in the Shared VPC networks, described in Section 7.3, is currently being used by Private Service Networking in this release. -## [1.x](https://github.com/terraform-google-modules/terraform-example-foundation/releases/tag/v1.0.0) +## [1.x](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/releases/tag/v1.0.0) ### Code Discrepancies #### Labeling diff --git a/README.md b/README.md index f7ff9a68..34c9fd79 100644 --- a/README.md +++ b/README.md @@ -2,9 +2,9 @@ ## Overview -This repository serves as a example for configuring an environment for the development and deployment of Machine Learning applications using the Vertex AI platform on Google Cloud. It seamlessly integrates the Cloud Foundation Toolkit (CFT) and implements robust security measures, drawing heavily from the [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation/tree/v4.0.0) codebase. +This repository serves as a example for configuring an environment for the development and deployment of Machine Learning applications using the Vertex AI platform on Google Cloud. It seamlessly integrates the Cloud Foundation Toolkit (CFT) and implements robust security measures, drawing heavily from the [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/tree/v4.0.0) codebase. -The repo is separated in distinct Terraform projects, each within their own directory that must be applied separately, but in sequence, for more information about each step, please refer to [terraform-example-foundation](https://github.com/terraform-google-modules/terraform-example-foundation/tree/v4.0.0). Comparing to the foundation repository, the key differences from the steps in foundation to steps in these repository are: +The repo is separated in distinct Terraform projects, each within their own directory that must be applied separately, but in sequence, for more information about each step, please refer to [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/tree/v4.0.0). Comparing to the foundation repository, the key differences from the steps in foundation to steps in these repository are: * [1. org](./1-org/) * Specific to this repository, it will also configure Machine Learning Organization Policies. diff --git a/docs/FAQ.md b/docs/FAQ.md index 82ab2d93..319c8d64 100644 --- a/docs/FAQ.md +++ b/docs/FAQ.md @@ -13,7 +13,7 @@ If you see other quota errors, see the [Quota documentation](https://cloud.googl ## What is a "named" branch? -Certain branches in the terraform-example-foundation are considered to be +Certain branches in the terraform-google-enterprise-genai are considered to be _named branches_. Pushing to a named branch causes the _apply_ command to be run. Pushing to branches other than the named branches does not run _apply_. diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index 7c3c025d..8f4637a4 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -191,7 +191,7 @@ At this time the alternatives are: 1. To use a [workaround](https://stackoverflow.com/a/62827358) to force Google API calls in Cloud Shell to use an IP from the `private.googleapis.com` range (199.36.153.8/30 ) or 1. To deploy the foundation code from a local machine that supports IPv6. -If you use the workaround, the API list should include the ones that are [allowed](../policy-library/policies/constraints/serviceusage_allow_basic_apis.yaml) in the terraform-example-foundation policy library. +If you use the workaround, the API list should include the ones that are [allowed](../policy-library/policies/constraints/serviceusage_allow_basic_apis.yaml) in the terraform-google-enterprise-genai policy library. ### Error: Unsupported attribute @@ -473,4 +473,4 @@ You can get this information from step `0-bootstrap` by running the following co **Terraform State lock possible causes:** -- If you realize that the Terraform State lock was due to a build timeout increase the build timeout on [build configuration](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/build/cloudbuild-tf-apply.yaml#L15). +- If you realize that the Terraform State lock was due to a build timeout increase the build timeout on [build configuration](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/build/cloudbuild-tf-apply.yaml#L15). diff --git a/docs/upgrading_to_v2.0.md b/docs/upgrading_to_v2.0.md index 6b92ea9e..c4db74b3 100644 --- a/docs/upgrading_to_v2.0.md +++ b/docs/upgrading_to_v2.0.md @@ -2,7 +2,7 @@ Before moving forward with adopting components of V2, please review the list of breaking changes below. You can find a list of all changes in the -[Changelog](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/CHANGELOG.md). +[Changelog](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/CHANGELOG.md). **Note:** There is no in-place upgrade path from v1 to v2. diff --git a/docs/upgrading_to_v3.0.md b/docs/upgrading_to_v3.0.md index c40d4410..b5690425 100644 --- a/docs/upgrading_to_v3.0.md +++ b/docs/upgrading_to_v3.0.md @@ -1,5 +1,5 @@ # Upgrade Guidance -Before moving forward with adopting components of v3, review the list of breaking changes below. You can find a complete list of features, bug fixes and other updates in the [Changelog](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/CHANGELOG.md). +Before moving forward with adopting components of v3, review the list of breaking changes below. You can find a complete list of features, bug fixes and other updates in the [Changelog](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/CHANGELOG.md). **Important:** There is no in-place upgrade path from v2 to v3. @@ -17,7 +17,7 @@ There is no direct path for upgrading from v2 to v3 as this may result in resour In case you require to integrate some of the v3's features, we recommend to review the documentation regarding the feature you are interested in and use v3's code as a guidance for its implementation. We also recommend to review the output from `terraform plan` for any destructive operations before applying the updates. -**Note:** You must verify that you are using the correct version for `terraform` and `gcloud`. You can check these and other additional requirements using this [validate script](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/scripts/validate-requirements.sh). +**Note:** You must verify that you are using the correct version for `terraform` and `gcloud`. You can check these and other additional requirements using this [validate script](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/scripts/validate-requirements.sh). ### Move Blocks diff --git a/helpers/foundation-deployer/README.md b/helpers/foundation-deployer/README.md index e8c54146..22c06132 100644 --- a/helpers/foundation-deployer/README.md +++ b/helpers/foundation-deployer/README.md @@ -29,11 +29,11 @@ Helper tool to deploy the Terraform example foundation. ### Prepare the deploy environment - Create a directory in the file system to host the Cloud Source repositories the will be created and a copy of the terraform example foundation. -- Clone the `terraform-example-foundation` repository on this directory. +- Clone the `terraform-google-enterprise-genai` repository on this directory. ```text deploy-directory/ - └── terraform-example-foundation + └── terraform-google-enterprise-genai ``` - Copy the file [global.tfvars.example](./global.tfvars.example) as `global.tfvars` to the same directory. @@ -41,21 +41,21 @@ Helper tool to deploy the Terraform example foundation. ```text deploy-directory/ └── global.tfvars - └── terraform-example-foundation + └── terraform-google-enterprise-genai ``` - Update `global.tfvars` with values from your environment. -- The `0-bootstrap` README [prerequisites](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#prerequisites) section has additional prerequisites needed to run this helper. +- The `0-bootstrap` README [prerequisites](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README.md#prerequisites) section has additional prerequisites needed to run this helper. - Variable `code_checkout_path` is the full path to `deploy-directory` directory. -- Variable `foundation_code_path` is the full path to `terraform-example-foundation` directory. +- Variable `foundation_code_path` is the full path to `terraform-google-enterprise-genai` directory. - See the READMEs for the stages for additional information: - - [0-bootstrap](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md) - - [1-org](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/README.md) - - [2-environments](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/2-environments/README.md) - - [3-networks-dual-svpc](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-dual-svpc) - - [3-networks-hub-and-spoke](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke) - - [4-projects](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects) - - [5-app-infra](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/5-app-infra) + - [0-bootstrap](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README.md) + - [1-org](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/1-org/README.md) + - [2-environments](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/2-environments/README.md) + - [3-networks-dual-svpc](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-dual-svpc) + - [3-networks-hub-and-spoke](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-hub-and-spoke) + - [4-projects](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/4-projects) + - [5-app-infra](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/5-app-infra) ### Location @@ -132,7 +132,7 @@ Im addition to the variables declared in the file `global.tfvars` for configurin └── gcp-policies-app-infra └── gcp-projects └── global.tfvars - └── terraform-example-foundation + └── terraform-google-enterprise-genai ``` ### Supported flags diff --git a/helpers/foundation-deployer/gcp/gcp.go b/helpers/foundation-deployer/gcp/gcp.go index c1671683..019922c2 100644 --- a/helpers/foundation-deployer/gcp/gcp.go +++ b/helpers/foundation-deployer/gcp/gcp.go @@ -23,7 +23,7 @@ import ( "github.com/mitchellh/go-testing-interface" "github.com/tidwall/gjson" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) const ( diff --git a/helpers/foundation-deployer/global.tfvars.example b/helpers/foundation-deployer/global.tfvars.example index 16867765..5da7ab8b 100644 --- a/helpers/foundation-deployer/global.tfvars.example +++ b/helpers/foundation-deployer/global.tfvars.example @@ -29,7 +29,7 @@ validator_project_id = "EXISTING_PROJECT_ID" // 0-bootstrap inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README.md#inputs org_id = "REPLACE_ME" # format "000000000000" billing_account = "REPLACE_ME" # format "000000-000000-000000" @@ -55,7 +55,7 @@ folder_prefix = "fldr" // Optional - for enabling the automatic groups creation, uncomment the groups // variable and update the values with the desired group names -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/0-bootstrap/README.md#optional---automatic-creation-of-google-cloud-identity-groups +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/0-bootstrap/README.md#optional---automatic-creation-of-google-cloud-identity-groups // After deploy, the Bootstrap service account will need to be granted "Group Admin" role in the // Google Workspace by a Super Admin before Cloud Build builds can be executed by the Bootstrap workspace. @@ -85,7 +85,7 @@ folder_prefix = "fldr" // 1-org inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/1-org/envs/shared/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/1-org/envs/shared/README.md#inputs audit_data_users = "REPLACE_ME" # "gcp-security-admins@example.com" billing_data_users = "REPLACE_ME" # "gcp-billing-data-users@example.com" @@ -101,8 +101,8 @@ log_export_storage_location = "US" billing_export_dataset_location = "US" // Choose witch network architecture to use: -// Dual Shared VPC: https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-dual-svpc/README.md -// Hub And Spoke: https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/README.md +// Dual Shared VPC: https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-dual-svpc/README.md +// Hub And Spoke: https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-hub-and-spoke/README.md enable_hub_and_spoke = false @@ -115,14 +115,14 @@ create_unique_tag_key = false // 2-environments inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/2-environments/envs/production/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/2-environments/envs/production/README.md#inputs monitoring_workspace_users = "REPLACE_ME" # "gcp-monitoring-admins@example.com" // 3-networks inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/envs/production/README.md#inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/3-networks-hub-and-spoke/envs/shared/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-hub-and-spoke/envs/production/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/3-networks-hub-and-spoke/envs/shared/README.md#inputs domain = "example.com." # The DNS name of peering managed zone. Must end with a period. @@ -149,7 +149,7 @@ target_name_server_addresses = [ // 4-projects inputs -// https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/4-projects/business_unit_1/production/README.md#inputs +// https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/4-projects/business_unit_1/production/README.md#inputs projects_gcs_location = "US" projects_kms_location = "us" diff --git a/helpers/foundation-deployer/go.mod b/helpers/foundation-deployer/go.mod index cd82def4..ff86ed66 100644 --- a/helpers/foundation-deployer/go.mod +++ b/helpers/foundation-deployer/go.mod @@ -1,4 +1,4 @@ -module github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer +module github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer go 1.20 @@ -8,7 +8,7 @@ require ( github.com/hashicorp/hcl/v2 v2.16.1 github.com/mitchellh/go-testing-interface v1.14.2-0.20210821155943-2d9075ca8770 github.com/stretchr/testify v1.8.2 - github.com/terraform-google-modules/terraform-example-foundation/test/integration v0.0.0-20230503230051-e9e2618ef515 + github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration v0.0.0-20230503230051-e9e2618ef515 github.com/tidwall/gjson v1.14.4 ) diff --git a/helpers/foundation-deployer/go.sum b/helpers/foundation-deployer/go.sum index b8f47f5f..e454ea3b 100644 --- a/helpers/foundation-deployer/go.sum +++ b/helpers/foundation-deployer/go.sum @@ -447,8 +447,8 @@ github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ github.com/stretchr/testify v1.8.2 h1:+h33VjcLVPDHtOdpUCuF+7gSuG3yGIftsP1YvFihtJ8= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= github.com/stretchr/testify v1.8.2/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4= -github.com/terraform-google-modules/terraform-example-foundation/test/integration v0.0.0-20230503230051-e9e2618ef515 h1:9WpwfiGRUEX5e3qeC93M/TWxmVeSD8vz1B8c5FumdAE= -github.com/terraform-google-modules/terraform-example-foundation/test/integration v0.0.0-20230503230051-e9e2618ef515/go.mod h1:4EkFeYb9/Cjbqc4uynVvcmV8MCK7VK5dGuR2yD4r8EU= +github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration v0.0.0-20230503230051-e9e2618ef515 h1:9WpwfiGRUEX5e3qeC93M/TWxmVeSD8vz1B8c5FumdAE= +github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration v0.0.0-20230503230051-e9e2618ef515/go.mod h1:4EkFeYb9/Cjbqc4uynVvcmV8MCK7VK5dGuR2yD4r8EU= github.com/tidwall/gjson v1.14.2/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= github.com/tidwall/gjson v1.14.4 h1:uo0p8EbA09J7RQaflQ1aBRffTR7xedD2bcIVSYxLnkM= github.com/tidwall/gjson v1.14.4/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk= diff --git a/helpers/foundation-deployer/main.go b/helpers/foundation-deployer/main.go index ed682cfa..1475a645 100644 --- a/helpers/foundation-deployer/main.go +++ b/helpers/foundation-deployer/main.go @@ -25,11 +25,11 @@ import ( "github.com/mitchellh/go-testing-interface" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/gcp" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/msg" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/stages" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/steps" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/utils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/gcp" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/msg" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/stages" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/steps" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/utils" ) var ( diff --git a/helpers/foundation-deployer/msg/msg.go b/helpers/foundation-deployer/msg/msg.go index c2972be2..5667e140 100644 --- a/helpers/foundation-deployer/msg/msg.go +++ b/helpers/foundation-deployer/msg/msg.go @@ -23,11 +23,11 @@ import ( const ( size = 70 - readmeURL = "https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/%s/README.md" + readmeURL = "https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/%s/README.md" cloudBuildURL = "https://console.cloud.google.com/cloud-build/builds;region=%s?project=%s" buildErrorURL = "https://console.cloud.google.com/cloud-build/builds;region=%s/%s?project=%s" quotaURL = "https://support.google.com/code/contact/billing_quota_increase" - troubleQuotaURL = "https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#billing-quota-exceeded" + troubleQuotaURL = "https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/docs/TROUBLESHOOTING.md#billing-quota-exceeded" groupAdminURL = "https://cloud.google.com/identity/docs/how-to/setup#assigning_an_admin_role_to_the_service_account" ) diff --git a/helpers/foundation-deployer/stages/apply.go b/helpers/foundation-deployer/stages/apply.go index 1afaf7cd..f78486e7 100644 --- a/helpers/foundation-deployer/stages/apply.go +++ b/helpers/foundation-deployer/stages/apply.go @@ -22,12 +22,12 @@ import ( "github.com/gruntwork-io/terratest/modules/terraform" "github.com/mitchellh/go-testing-interface" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/gcp" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/msg" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/steps" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/utils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/gcp" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/msg" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/steps" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/utils" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func DeployBootstrapStage(t testing.TB, s steps.Steps, tfvars GlobalTFVars, c CommonConf) error { diff --git a/helpers/foundation-deployer/stages/data.go b/helpers/foundation-deployer/stages/data.go index 67d4e7be..59f90df2 100644 --- a/helpers/foundation-deployer/stages/data.go +++ b/helpers/foundation-deployer/stages/data.go @@ -24,7 +24,7 @@ import ( "github.com/gruntwork-io/terratest/modules/terraform" "github.com/mitchellh/go-testing-interface" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/utils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/utils" ) const ( diff --git a/helpers/foundation-deployer/stages/destroy.go b/helpers/foundation-deployer/stages/destroy.go index f9de4880..250dc512 100644 --- a/helpers/foundation-deployer/stages/destroy.go +++ b/helpers/foundation-deployer/stages/destroy.go @@ -23,9 +23,9 @@ import ( "github.com/gruntwork-io/terratest/modules/terraform" "github.com/mitchellh/go-testing-interface" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/steps" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/utils" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/steps" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/utils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) const ( @@ -194,9 +194,9 @@ func destroyStage(t testing.TB, sc StageConf, s steps.Steps, c CommonConf) error for _, g := range groupingUnits { err := s.RunDestroyStep(fmt.Sprintf("%s.%s.apply-shared", sc.Repo, g), func() error { options := &terraform.Options{ - TerraformDir: filepath.Join(gcpPath, g, "shared"), - Logger: c.Logger, - NoColor: true, + TerraformDir: filepath.Join(gcpPath, g, "shared"), + Logger: c.Logger, + NoColor: true, RetryableTerraformErrors: testutils.RetryableTransientErrors, MaxRetries: 2, TimeBetweenRetries: 2 * time.Minute, diff --git a/helpers/foundation-deployer/stages/validate.go b/helpers/foundation-deployer/stages/validate.go index 89851b40..af292e75 100644 --- a/helpers/foundation-deployer/stages/validate.go +++ b/helpers/foundation-deployer/stages/validate.go @@ -21,7 +21,7 @@ import ( "github.com/mitchellh/go-testing-interface" - "github.com/terraform-google-modules/terraform-example-foundation/helpers/foundation-deployer/gcp" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/helpers/foundation-deployer/gcp" ) const ( diff --git a/scripts/set-tfc-backend-and-remote.sh b/scripts/set-tfc-backend-and-remote.sh index 5da9ca56..087e8344 100755 --- a/scripts/set-tfc-backend-and-remote.sh +++ b/scripts/set-tfc-backend-and-remote.sh @@ -15,7 +15,7 @@ # limitations under the License. # Define the root folder where you want to start the search and renaming -root_folder="./terraform-example-foundation" +root_folder="./terraform-google-enterprise-genai" # Use 'find' to locate all files named 'backend.tf' # in the specified folder and its subfolders, excluding directories that start with ".terraform" diff --git a/scripts/validate-requirements.sh b/scripts/validate-requirements.sh index 1acca7de..b3d11889 100755 --- a/scripts/validate-requirements.sh +++ b/scripts/validate-requirements.sh @@ -21,7 +21,7 @@ # Expected versions of the installers TF_VERSION="1.3.0" # Version 393.0.0 due to terraform-tools 0.5.0 version that fixes the issue -# mentioned in this PR https://github.com/terraform-google-modules/terraform-example-foundation/pull/729#discussion_r919427668 +# mentioned in this PR https://github.com/terraform-google-modules/terraform-google-enterprise-genai/pull/729#discussion_r919427668 GCLOUD_SDK_VERSION="393.0.0" GIT_VERSION="2.28.0" @@ -126,7 +126,7 @@ function validate_git(){ if ! git config init.defaultBranch | grep "main" >/dev/null ; then echo " git default branch must be configured as main." - echo " See the instructions at https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting ." + echo " See the instructions at https://github.com/terraform-google-modules/terraform-google-enterprise-genai/blob/master/docs/TROUBLESHOOTING.md#default-branch-setting ." ERRORS+=$' git default branch must be configured as main.\n' fi } diff --git a/test/integration/bootstrap/bootstrap_test.go b/test/integration/bootstrap/bootstrap_test.go index 3ae75956..b4e9dc3f 100644 --- a/test/integration/bootstrap/bootstrap_test.go +++ b/test/integration/bootstrap/bootstrap_test.go @@ -29,7 +29,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) // fileExists check if a give file exists @@ -47,7 +47,7 @@ func fileExists(filePath string) (bool, error) { func TestBootstrap(t *testing.T) { vars := map[string]interface{}{ - "bucket_force_destroy": true, + "bucket_force_destroy": true, "bucket_tfstate_kms_force_destroy": true, } diff --git a/test/integration/envs/envs_test.go b/test/integration/envs/envs_test.go index 8330cb10..251adf8f 100644 --- a/test/integration/envs/envs_test.go +++ b/test/integration/envs/envs_test.go @@ -24,7 +24,7 @@ import ( "github.com/GoogleCloudPlatform/cloud-foundation-toolkit/infra/blueprint-test/pkg/utils" "github.com/stretchr/testify/assert" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func TestEnvs(t *testing.T) { diff --git a/test/integration/go.mod b/test/integration/go.mod index 999f0aba..1c5a4552 100644 --- a/test/integration/go.mod +++ b/test/integration/go.mod @@ -1,4 +1,4 @@ -module github.com/terraform-google-modules/terraform-example-foundation/test/integration +module github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration go 1.20 diff --git a/test/integration/networks/networks_test.go b/test/integration/networks/networks_test.go index e4c1d5d3..784c8888 100644 --- a/test/integration/networks/networks_test.go +++ b/test/integration/networks/networks_test.go @@ -26,7 +26,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func getNetworkMode(t *testing.T) string { diff --git a/test/integration/org/org_test.go b/test/integration/org/org_test.go index b0ba5665..e3717412 100644 --- a/test/integration/org/org_test.go +++ b/test/integration/org/org_test.go @@ -27,7 +27,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func TestOrg(t *testing.T) { diff --git a/test/integration/projects-shared/projects_shared_test.go b/test/integration/projects-shared/projects_shared_test.go index 67454134..f25f47c1 100644 --- a/test/integration/projects-shared/projects_shared_test.go +++ b/test/integration/projects-shared/projects_shared_test.go @@ -25,7 +25,7 @@ import ( "github.com/gruntwork-io/terratest/modules/terraform" "github.com/stretchr/testify/assert" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func TestProjectsShared(t *testing.T) { diff --git a/test/integration/projects/projects_test.go b/test/integration/projects/projects_test.go index d8bc2f75..e7ecd2db 100644 --- a/test/integration/projects/projects_test.go +++ b/test/integration/projects/projects_test.go @@ -27,7 +27,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func getNetworkMode(t *testing.T) string { diff --git a/test/integration/shared/shared_test.go b/test/integration/shared/shared_test.go index f865c722..e21b4e0a 100644 --- a/test/integration/shared/shared_test.go +++ b/test/integration/shared/shared_test.go @@ -26,7 +26,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" - "github.com/terraform-google-modules/terraform-example-foundation/test/integration/testutils" + "github.com/terraform-google-modules/terraform-google-enterprise-genai/test/integration/testutils" ) func isHubAndSpokeMode(t *testing.T) bool { From 454bee48bc5389080220006d5f6c461517e30550 Mon Sep 17 00:00:00 2001 From: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> Date: Tue, 11 Jun 2024 16:07:49 -0300 Subject: [PATCH 2/4] chore(docs): create deployment steps (1-4) on top of foundation (#43) * first commit * add more info * rename composer and vertex pipe * add ml pipe * docs: update deploy on foundation * update image path * add space * correct sa * update image path * edit readme * change references from 7- * revert some changes * reverting changes * add policies * reformat code block * add some modifications to network * add 4-projects * update dates * adjust headings * fixing some lint issues * updating docs * remove duplicate headings * add projects envs * adjust 1-org step * fix type * update some 2-env * some adjustments on 3-network * update outputs on 2-env * update networks * updating 4-projects * update 4-projects steps after test * update readme * adjust some formatting * rename ml_infra_project -> ml_infra_projects * remove other text * rename repo * update non-prod -> nonprod * renaming non-prod -> nonprod * restore files --- .github/workflows/lint.yaml | 4 +- Makefile | 2 +- README.md | 2 + docs/assets/terraform/1-org/README.md | 1 + docs/assets/terraform/1-org/ml_key_rings.tf | 10 + .../terraform/1-org/ml_ops_org_policy.tf | 67 + .../assets/terraform/2-environments/README.md | 1 + .../terraform/2-environments/ml_key_rings.tf | 33 + .../terraform/2-environments/ml_logging.tf | 87 + .../3-networks-dual-svpc/ml_dns_notebooks.tf | 29 + .../4-projects/common.auto.example.tfvars | 17 + .../development.auto.example.tfvars | 19 + .../ml_business_unit/development/README.md | 25 + .../ml_business_unit/development/backend.tf | 22 + .../development/backend.tf.cloud.example | 23 + .../development/common.auto.tfvars | 1 + .../development/development.auto.tfvars | 1 + .../ml_business_unit/development/locals.tf | 20 +- .../ml_business_unit/development/main.tf | 35 + .../ml_business_unit/development/outputs.tf | 35 + .../ml_business_unit/development/remote.tf | 60 + .../ml_business_unit/development/variables.tf | 78 + .../ml_business_unit/development/versions.tf | 30 + .../ml_business_unit/nonproduction/README.md | 25 + .../ml_business_unit/nonproduction/backend.tf | 22 + .../nonproduction/backend.tf.cloud.example | 23 + .../nonproduction/common.auto.tfvars | 1 + .../ml_business_unit/nonproduction/locals.tf | 19 + .../ml_business_unit/nonproduction/main.tf | 34 + .../nonproduction/non-production.auto.tfvars | 1 + .../ml_business_unit/nonproduction/outputs.tf | 36 + .../nonproduction/outputs.tf.backup | 125 + .../ml_business_unit/nonproduction/remote.tf | 60 + .../nonproduction/variables.tf | 78 + .../ml_business_unit/production/README.md | 24 + .../ml_business_unit/production/backend.tf | 22 + .../production/backend.tf.cloud.example | 23 + .../production/common.auto.tfvars | 1 + .../ml_business_unit/production/locals.tf | 19 + .../ml_business_unit/production/main.tf | 34 + .../ml_business_unit/production/outputs.tf | 35 + .../production/outputs.tf.backup | 125 + .../production/production.auto.tfvars | 1 + .../ml_business_unit/production/remote.tf | 60 + .../ml_business_unit/production/variables.tf | 59 + .../ml_business_unit/shared/README.md | 40 + .../ml_business_unit/shared/backend.tf | 22 + .../shared/backend.tf.cloud.example | 23 + .../shared/common.auto.tfvars | 1 + .../shared/example_infra_pipeline.tf | 90 + .../shared/ml_infra_projects.tf | 36 + .../ml_business_unit/shared/outputs.tf | 94 + .../ml_business_unit/shared/remote.tf | 51 + .../shared/remote.tf.cloud.example | 40 + .../shared/shared.auto.tfvars | 1 + .../ml_business_unit/shared/variables.tf | 95 + .../ml_business_unit/shared/versions.tf | 30 + .../env_folders/business_unit_folder.tf | 24 + .../4-projects/modules/env_folders/outputs.tf | 20 + .../4-projects/modules/env_folders/remote.tf | 29 + .../modules/env_folders/variables.tf | 36 + .../modules/env_folders/versions.tf | 29 + .../modules/ml_env/example_vertex.tf | 240 ++ .../4-projects/modules/ml_env/outputs.tf | 30 + .../4-projects/modules/ml_env/pubsub.tf | 34 + .../4-projects/modules/ml_env/remote.tf | 91 + .../modules/ml_env/remote.tf.cloud.example | 52 + .../4-projects/modules/ml_env/secret.tf | 62 + .../4-projects/modules/ml_env/variables.tf | 101 + .../4-projects/modules/ml_env/versions.tf | 33 + .../ml_infra_projects/artifacts_project.tf | 87 + .../modules/ml_infra_projects/locals.tf | 19 + .../modules/ml_infra_projects/outputs.tf | 45 + .../service_catalog_project.tf | 124 + .../modules/ml_infra_projects/variables.tf | 203 ++ .../modules/ml_infra_projects/versions.tf | 20 + .../4-projects/modules/ml_kms_key/main.tf | 43 + .../4-projects/modules/ml_kms_key/outputs.tf | 20 + .../modules/ml_kms_key/variables.tf | 36 + .../4-projects/modules/ml_kms_key/versions.tf | 19 + .../modules/ml_single_project/README.md | 52 + .../modules/ml_single_project/main.tf | 141 ++ .../modules/ml_single_project/outputs.tf | 45 + .../modules/ml_single_project/variables.tf | 187 ++ .../modules/ml_single_project/versions.tf | 29 + .../non-production.auto.example.tfvars | 19 + .../4-projects/production.auto.example.tfvars | 19 + .../4-projects/shared.auto.example.tfvars | 22 + docs/deploy_on_foundation_v4.0.0.md | 2166 +++++++++++++++++ 89 files changed, 6012 insertions(+), 17 deletions(-) create mode 100644 docs/assets/terraform/1-org/README.md create mode 100644 docs/assets/terraform/1-org/ml_key_rings.tf create mode 100644 docs/assets/terraform/1-org/ml_ops_org_policy.tf create mode 100644 docs/assets/terraform/2-environments/README.md create mode 100644 docs/assets/terraform/2-environments/ml_key_rings.tf create mode 100644 docs/assets/terraform/2-environments/ml_logging.tf create mode 100644 docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf create mode 100644 docs/assets/terraform/4-projects/common.auto.example.tfvars create mode 100644 docs/assets/terraform/4-projects/development.auto.example.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/README.md create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf.cloud.example create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/development/common.auto.tfvars create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/development/development.auto.tfvars rename policy-library/storage_logging.yaml => docs/assets/terraform/4-projects/ml_business_unit/development/locals.tf (58%) create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/main.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/outputs.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/remote.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/variables.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/development/versions.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/README.md create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf.cloud.example create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/common.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/locals.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/main.tf create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/non-production.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf.backup create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/remote.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/nonproduction/variables.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/README.md create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf.cloud.example create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/production/common.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/locals.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/main.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf.backup create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/production/production.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/remote.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/production/variables.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/README.md create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf.cloud.example create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/shared/common.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/example_infra_pipeline.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/ml_infra_projects.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/outputs.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf.cloud.example create mode 120000 docs/assets/terraform/4-projects/ml_business_unit/shared/shared.auto.tfvars create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/variables.tf create mode 100644 docs/assets/terraform/4-projects/ml_business_unit/shared/versions.tf create mode 100644 docs/assets/terraform/4-projects/modules/env_folders/business_unit_folder.tf create mode 100644 docs/assets/terraform/4-projects/modules/env_folders/outputs.tf create mode 100644 docs/assets/terraform/4-projects/modules/env_folders/remote.tf create mode 100644 docs/assets/terraform/4-projects/modules/env_folders/variables.tf create mode 100644 docs/assets/terraform/4-projects/modules/env_folders/versions.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/example_vertex.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/outputs.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/pubsub.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/remote.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/remote.tf.cloud.example create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/secret.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/variables.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_env/versions.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/artifacts_project.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/locals.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/outputs.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/service_catalog_project.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/variables.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_infra_projects/versions.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_kms_key/main.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_kms_key/outputs.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_kms_key/variables.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_kms_key/versions.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_single_project/README.md create mode 100644 docs/assets/terraform/4-projects/modules/ml_single_project/main.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_single_project/outputs.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_single_project/variables.tf create mode 100644 docs/assets/terraform/4-projects/modules/ml_single_project/versions.tf create mode 100644 docs/assets/terraform/4-projects/non-production.auto.example.tfvars create mode 100644 docs/assets/terraform/4-projects/production.auto.example.tfvars create mode 100644 docs/assets/terraform/4-projects/shared.auto.example.tfvars create mode 100644 docs/deploy_on_foundation_v4.0.0.md diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 934f89f6..0b89cae9 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -48,10 +48,10 @@ jobs: env: DISABLE_TFLINT: 1 ENABLE_PARALLEL: 0 - EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning + EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning|\./docs/assets/terraform - run: docker run --rm -e DISABLE_TFLINT -e ENABLE_PARALLEL -e EXCLUDE_LINT_DIRS -v ${{ github.workspace }}:/workspace ${{ steps.variables.outputs.dev-tools }} /usr/local/bin/test_lint.sh env: DISABLE_TFLINT: 1 ENABLE_PARALLEL: 0 - EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning + EXCLUDE_LINT_DIRS: \./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning|\./docs/assets/terraform diff --git a/Makefile b/Makefile index f315bc4f..6ca20792 100644 --- a/Makefile +++ b/Makefile @@ -28,7 +28,7 @@ docker_test_lint: docker run --rm -it \ -e ENABLE_PARALLEL=0 \ -e DISABLE_TFLINT=1 \ - -e EXCLUDE_LINT_DIRS="\./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning" \ + -e EXCLUDE_LINT_DIRS="\./5-app-infra/6-machine-learning|\./5-app-infra/projects/machine-learning|\./docs/assets/terraform" \ -v $(CURDIR):/workspace \ $(REGISTRY_URL)/${DOCKER_IMAGE_DEVELOPER_TOOLS}:${DOCKER_TAG_VERSION_DEVELOPER_TOOLS} \ /usr/local/bin/test_lint.sh diff --git a/README.md b/README.md index 34c9fd79..de72bcb6 100644 --- a/README.md +++ b/README.md @@ -6,6 +6,8 @@ This repository serves as a example for configuring an environment for the devel The repo is separated in distinct Terraform projects, each within their own directory that must be applied separately, but in sequence, for more information about each step, please refer to [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/tree/v4.0.0). Comparing to the foundation repository, the key differences from the steps in foundation to steps in these repository are: +Alternatively, the user can follow steps on [`docs/deploy_on_foundation_v4.0.0.md`](./docs/deploy_on_foundation_v4.0.0.md), to deploy the codebase on top of an existing foundation instance. + * [1. org](./1-org/) * Specific to this repository, it will also configure Machine Learning Organization Policies. * [2. environments](./2-environments/) diff --git a/docs/assets/terraform/1-org/README.md b/docs/assets/terraform/1-org/README.md new file mode 100644 index 00000000..092ada1b --- /dev/null +++ b/docs/assets/terraform/1-org/README.md @@ -0,0 +1 @@ +# Terraform code in this directory is used on docs as reference/assets. diff --git a/docs/assets/terraform/1-org/ml_key_rings.tf b/docs/assets/terraform/1-org/ml_key_rings.tf new file mode 100644 index 00000000..9e3f6304 --- /dev/null +++ b/docs/assets/terraform/1-org/ml_key_rings.tf @@ -0,0 +1,10 @@ +module "kms_keyring" { + source = "../../modules/ml_kms_keyring" + + keyring_admins = [ + "serviceAccount:${local.projects_step_terraform_service_account_email}" + ] + project_id = module.org_kms.project_id + keyring_regions = var.keyring_regions + keyring_name = var.keyring_name +} diff --git a/docs/assets/terraform/1-org/ml_ops_org_policy.tf b/docs/assets/terraform/1-org/ml_ops_org_policy.tf new file mode 100644 index 00000000..4fa82aa7 --- /dev/null +++ b/docs/assets/terraform/1-org/ml_ops_org_policy.tf @@ -0,0 +1,67 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "ml_organization_policies" { + source = "../../modules/ml-org-policies" + + org_id = local.organization_id + folder_id = local.folder_id + + allowed_locations = [ + "in:us-locations" + ] + + allowed_vertex_vpc_networks = { + parent_type = "project" + ids = [for instance in module.base_restricted_environment_network : instance.restricted_shared_vpc_project_id], + } + + allowed_vertex_images = [ + "ainotebooks-vm/deeplearning-platform-release/image-family/pytorch-1-13-cu113-notebooks", + "ainotebooks-vm/deeplearning-platform-release/image-family/pytorch-1-13-cu113-notebooks", + "ainotebooks-vm/deeplearning-platform-release/image-family/common-cu113-notebooks", + "ainotebooks-vm/deeplearning-platform-release/image-family/common-cpu-notebooks", + "ainotebooks-container/us-docker.pkg.dev/deeplearning-platform-release/gcr.io/base-cu113.py310", + "ainotebooks-container/us-docker.pkg.dev/deeplearning-platform-release/gcr.io/base-cu113.py37", + "ainotebooks-container/us-docker.pkg.dev/deeplearning-platform-release/gcr.io/base-cu110.py310", + "ainotebooks-container/us-docker.pkg.dev/deeplearning-platform-release/gcr.io/tf2-cpu.2-12.py310", + "ainotebooks-container/us-docker.pkg.dev/deeplearning-platform-release/gcr.io/tf2-gpu.2-12.py310" + ] + + restricted_services = [ + "alloydb.googleapis.com" + ] + + allowed_integrations = [ + "github.com", + "source.developers.google.com" + ] + + restricted_tls_versions = [ + "TLS_VERSION_1", + "TLS_VERSION_1_1" + ] + + restricted_non_cmek_services = [ + "bigquery.googleapis.com", + "aiplatform.googleapis.com" + ] + + allowed_vertex_access_modes = [ + "single-user", + "service-account" + ] +} diff --git a/docs/assets/terraform/2-environments/README.md b/docs/assets/terraform/2-environments/README.md new file mode 100644 index 00000000..092ada1b --- /dev/null +++ b/docs/assets/terraform/2-environments/README.md @@ -0,0 +1 @@ +# Terraform code in this directory is used on docs as reference/assets. diff --git a/docs/assets/terraform/2-environments/ml_key_rings.tf b/docs/assets/terraform/2-environments/ml_key_rings.tf new file mode 100644 index 00000000..05abada3 --- /dev/null +++ b/docs/assets/terraform/2-environments/ml_key_rings.tf @@ -0,0 +1,33 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + logging_key_name = module.env_logs.project_id +} + +// Creates a keyring with logging key for each region (us-central1, us-east4) +module "kms_keyring" { + source = "../ml_kms_keyring" + + keyring_admins = [ + "serviceAccount:${local.projects_step_terraform_service_account_email}" + ] + project_id = module.env_kms.project_id + keyring_regions = var.keyring_regions + keyring_name = var.keyring_name + keys = [local.logging_key_name] + kms_prevent_destroy = var.kms_prevent_destroy +} diff --git a/docs/assets/terraform/2-environments/ml_logging.tf b/docs/assets/terraform/2-environments/ml_logging.tf new file mode 100644 index 00000000..61a4e0e5 --- /dev/null +++ b/docs/assets/terraform/2-environments/ml_logging.tf @@ -0,0 +1,87 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +data "google_storage_project_service_account" "gcs_logging_account" { + project = module.env_logs.project_id +} + +/****************************************** + Project for Environment Logging +*****************************************/ + +module "env_logs" { + source = "terraform-google-modules/project-factory/google" + version = "~> 14.0" + + random_project_id = true + random_project_id_length = 4 + default_service_account = "deprivilege" + name = "${local.project_prefix}-${var.environment_code}-logging" + org_id = local.org_id + billing_account = local.billing_account + folder_id = google_folder.env.id + activate_apis = ["logging.googleapis.com", "billingbudgets.googleapis.com", "storage.googleapis.com"] + + labels = { + environment = var.env + application_name = "env-logging" + billing_code = "1234" + primary_contact = "example1" + secondary_contact = "example2" + business_code = "abcd" + env_code = var.environment_code + } + budget_alert_pubsub_topic = var.project_budget.logging_alert_pubsub_topic + budget_alert_spent_percents = var.project_budget.logging_alert_spent_percents + budget_amount = var.project_budget.logging_budget_amount + budget_alert_spend_basis = var.project_budget.logging_budget_alert_spend_basis + +} + +// Create Bucket for this project +resource "google_storage_bucket" "log_bucket" { + name = "${var.gcs_bucket_prefix}-${module.env_logs.project_id}" + location = var.gcs_logging_bucket_location + project = module.env_logs.project_id + uniform_bucket_level_access = true + + dynamic "retention_policy" { + for_each = var.gcs_logging_retention_period != null ? [var.gcs_logging_retention_period] : [] + content { + is_locked = var.gcs_logging_retention_period.is_locked + retention_period = var.gcs_logging_retention_period.retention_period_days * 24 * 60 * 60 + } + } + + encryption { + default_kms_key_name = google_kms_crypto_key_iam_member.gcs_logging_key.crypto_key_id #module.kms_keyring.keys_by_region[var.gcs_logging_bucket_location][local.logging_key_name] + } +} + +/****************************************** + Logging Bucket - IAM +*****************************************/ +# resource "google_storage_bucket_iam_member" "bucket_logging" { +# bucket = google_storage_bucket.log_bucket.name +# role = "roles/storage.objectCreator" +# member = "group:cloud-storage-analytics@google.com" +# } + +resource "google_kms_crypto_key_iam_member" "gcs_logging_key" { + crypto_key_id = module.kms_keyring.keys_by_region[var.gcs_logging_bucket_location][local.logging_key_name] + role = "roles/cloudkms.cryptoKeyEncrypterDecrypter" + member = "serviceAccount:${data.google_storage_project_service_account.gcs_logging_account.email_address}" +} diff --git a/docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf b/docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf new file mode 100644 index 00000000..fdae1d35 --- /dev/null +++ b/docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf @@ -0,0 +1,29 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +module "ml_dns_vertex_ai" { + source = "../ml_dns_notebooks" + + project_id = local.restricted_project_id + private_service_connect_ip = var.restricted_private_service_connect_ip + private_visibility_config_networks = [module.restricted_shared_vpc.network_self_link] + zone_names = { + kernels_googleusercontent_zone = "dz-${var.environment_code}-shared-restricted-kernels-googleusercontent" + notebooks_googleusercontent_zone = "dz-${var.environment_code}-shared-restricted-notebooks-googleusercontent" + notebooks_cloudgoogle_zone = "dz-${var.environment_code}-shared-restricted-notebooks" + } +} \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/common.auto.example.tfvars b/docs/assets/terraform/4-projects/common.auto.example.tfvars new file mode 100644 index 00000000..49cf2d8a --- /dev/null +++ b/docs/assets/terraform/4-projects/common.auto.example.tfvars @@ -0,0 +1,17 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +remote_state_bucket = "REMOTE_STATE_BUCKET" diff --git a/docs/assets/terraform/4-projects/development.auto.example.tfvars b/docs/assets/terraform/4-projects/development.auto.example.tfvars new file mode 100644 index 00000000..b8e661cb --- /dev/null +++ b/docs/assets/terraform/4-projects/development.auto.example.tfvars @@ -0,0 +1,19 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +location_kms = "us" +location_gcs = "US" +env = "development" diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/README.md b/docs/assets/terraform/4-projects/ml_business_unit/development/README.md new file mode 100644 index 00000000..fd36be6e --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/README.md @@ -0,0 +1,25 @@ + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| default\_region | Default region to create resources where applicable. | `string` | `"us-central1"` | no | +| env | The environment this deployment belongs to (ie. development) | `string` | n/a | yes | +| key\_rotation\_period | Rotation period in seconds to be used for KMS Key | `string` | `"7776000s"` | no | +| location\_gcs | Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring) | `string` | `"US"` | no | +| location\_kms | Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket) | `string` | `"us"` | no | +| peering\_module\_depends\_on | List of modules or resources peering module depends on. | `list(any)` | `[]` | no | +| project\_budget | Budget configuration.
budget\_amount: The amount to use as the budget.
alert\_spent\_percents: A list of percentages of the budget to alert on when threshold is exceeded.
alert\_pubsub\_topic: The name of the Cloud Pub/Sub topic where budget related messages will be published, in the form of `projects/{project_id}/topics/{topic_id}`.
alert\_spend\_basis: The type of basis used to determine if spend has passed the threshold. Possible choices are `CURRENT_SPEND` or `FORECASTED_SPEND` (default). | 
object({
    budget_amount        = optional(number, 1000)
    alert_spent_percents = optional(list(number), [1.2])
    alert_pubsub_topic   = optional(string, null)
    alert_spend_basis    = optional(string, "FORECASTED_SPEND")
  })
| `{}` | no | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | +| tfc\_org\_name | Name of the TFC organization. | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| enable\_cloudbuild\_deploy | Enable infra deployment using Cloud Build. | +| machine\_learning\_kms\_keys | Key ID for the machine learning project. | +| machine\_learning\_project\_id | Project machine learning project. | +| machine\_learning\_project\_number | Project number of machine learning project. | + + diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf b/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf new file mode 100644 index 00000000..e5ccedf9 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf @@ -0,0 +1,22 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + backend "gcs" { + bucket = "UPDATE_PROJECTS_BACKEND" + prefix = "terraform/projects/ml_business_unit/development" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf.cloud.example b/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf.cloud.example new file mode 100644 index 00000000..c09ed6d6 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/backend.tf.cloud.example @@ -0,0 +1,23 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + cloud { + workspaces { + name = "4-ml-development" + } + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/common.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/development/common.auto.tfvars new file mode 120000 index 00000000..39aaa462 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/common.auto.tfvars @@ -0,0 +1 @@ +../../common.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/development.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/development/development.auto.tfvars new file mode 120000 index 00000000..0351835d --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/development.auto.tfvars @@ -0,0 +1 @@ +../../development.auto.tfvars \ No newline at end of file diff --git a/policy-library/storage_logging.yaml b/docs/assets/terraform/4-projects/ml_business_unit/development/locals.tf similarity index 58% rename from policy-library/storage_logging.yaml rename to docs/assets/terraform/4-projects/ml_business_unit/development/locals.tf index b1e003ef..b10bf71e 100644 --- a/policy-library/storage_logging.yaml +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/locals.tf @@ -1,4 +1,4 @@ -# Copyright 2021 Google LLC +# Copyright 2024 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -12,16 +12,8 @@ # See the License for the specific language governing permissions and # limitations under the License. # -apiVersion: constraints.gatekeeper.sh/v1alpha1 -kind: GCPStorageLoggingConstraintV1 -metadata: - name: storage_logging - annotations: - description: Ensure storage logs are delivered to a separate bucket -spec: - severity: high - match: - ancestries: - - "organizations/**" - excludedAncestries: [] # optional, default is no exclusions - parameters: {} +locals { + repo_name = "ml-composer" + business_code = "ml" + business_unit = "ml_business_unit" +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/main.tf b/docs/assets/terraform/4-projects/ml_business_unit/development/main.tf new file mode 100644 index 00000000..bfc78518 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/main.tf @@ -0,0 +1,35 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "bu_folder" { + source = "../../modules/env_folders" + business_code = local.business_code + remote_state_bucket = var.remote_state_bucket + env = var.env +} + +module "ml_env" { + source = "../../modules/ml_env" + + env = var.env + business_code = local.business_code + business_unit = local.business_unit + remote_state_bucket = var.remote_state_bucket + location_gcs = var.location_gcs + tfc_org_name = var.tfc_org_name + business_unit_folder = module.bu_folder.business_unit_folder +} + diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/outputs.tf b/docs/assets/terraform/4-projects/ml_business_unit/development/outputs.tf new file mode 100644 index 00000000..947cd723 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/outputs.tf @@ -0,0 +1,35 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "machine_learning_project_id" { + description = "Project machine learning project." + value = module.ml_env.machine_learning_project_id +} + +output "machine_learning_project_number" { + description = "Project number of machine learning project." + value = module.ml_env.machine_learning_project_number +} + +output "machine_learning_kms_keys" { + description = "Key ID for the machine learning project." + value = module.ml_env.machine_learning_kms_keys +} + +output "enable_cloudbuild_deploy" { + description = "Enable infra deployment using Cloud Build." + value = local.enable_cloudbuild_deploy +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/remote.tf b/docs/assets/terraform/4-projects/ml_business_unit/development/remote.tf new file mode 100644 index 00000000..19fd1ffb --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/remote.tf @@ -0,0 +1,60 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.terraform_remote_state.bootstrap.outputs.common_config.org_id + parent_folder = data.terraform_remote_state.bootstrap.outputs.common_config.parent_folder + parent = data.terraform_remote_state.bootstrap.outputs.common_config.parent_id + location_gcs = try(data.terraform_remote_state.bootstrap.outputs.common_config.default_region, var.location_gcs) + billing_account = data.terraform_remote_state.bootstrap.outputs.common_config.billing_account + common_folder_name = data.terraform_remote_state.org.outputs.common_folder_name + common_kms_project_id = data.terraform_remote_state.org.outputs.org_kms_project_id + default_region = data.terraform_remote_state.bootstrap.outputs.common_config.default_region + project_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.project_prefix + folder_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.folder_prefix + projects_remote_bucket_tfstate = data.terraform_remote_state.bootstrap.outputs.projects_gcs_bucket_tfstate + cloud_build_private_worker_pool_id = try(data.terraform_remote_state.bootstrap.outputs.cloud_build_private_worker_pool_id, "") + cloud_builder_artifact_repo = try(data.terraform_remote_state.bootstrap.outputs.cloud_builder_artifact_repo, "") + enable_cloudbuild_deploy = local.cloud_builder_artifact_repo != "" + environment_kms_key_ring = data.terraform_remote_state.environments_env.outputs.key_rings +} + +data "terraform_remote_state" "bootstrap" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/bootstrap/state" + } +} + +data "terraform_remote_state" "org" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/org/state" + } +} + +data "terraform_remote_state" "environments_env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/development/variables.tf b/docs/assets/terraform/4-projects/ml_business_unit/development/variables.tf new file mode 100644 index 00000000..59a87d1f --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/development/variables.tf @@ -0,0 +1,78 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "env" { + description = "The environment this deployment belongs to (ie. development)" + type = string +} +variable "default_region" { + description = "Default region to create resources where applicable." + type = string + default = "us-central1" +} + +variable "remote_state_bucket" { + description = "Backend bucket to load Terraform Remote State Data from previous steps." + type = string +} + +variable "location_kms" { + description = "Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket)" + type = string + default = "us" +} + +variable "location_gcs" { + description = "Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring)" + type = string + default = "US" +} + +variable "peering_module_depends_on" { + description = "List of modules or resources peering module depends on." + type = list(any) + default = [] +} + +variable "tfc_org_name" { + description = "Name of the TFC organization." + type = string + default = "" +} + +variable "project_budget" { + description = < +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| default\_region | Default region to create resources where applicable. | `string` | `"us-central1"` | no | +| env | The environment this deployment belongs to (ie. development) | `string` | n/a | yes | +| key\_rotation\_period | Rotation period in seconds to be used for KMS Key | `string` | `"7776000s"` | no | +| location\_gcs | Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring) | `string` | `"US"` | no | +| location\_kms | Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket) | `string` | `"us"` | no | +| peering\_module\_depends\_on | List of modules or resources peering module depends on. | `list(any)` | `[]` | no | +| project\_budget | Budget configuration.
budget\_amount: The amount to use as the budget.
alert\_spent\_percents: A list of percentages of the budget to alert on when threshold is exceeded.
alert\_pubsub\_topic: The name of the Cloud Pub/Sub topic where budget related messages will be published, in the form of `projects/{project_id}/topics/{topic_id}`.
alert\_spend\_basis: The type of basis used to determine if spend has passed the threshold. Possible choices are `CURRENT_SPEND` or `FORECASTED_SPEND` (default). | 
object({
    budget_amount        = optional(number, 1000)
    alert_spent_percents = optional(list(number), [1.2])
    alert_pubsub_topic   = optional(string, null)
    alert_spend_basis    = optional(string, "FORECASTED_SPEND")
  })
| `{}` | no | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | +| tfc\_org\_name | Name of the TFC organization. | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| enable\_cloudbuild\_deploy | Enable infra deployment using Cloud Build. | +| machine\_learning\_kms\_keys | Key ID for the machine learning project. | +| machine\_learning\_project\_id | Project machine learning project. | +| machine\_learning\_project\_number | Project number of machine learning project. | + + diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf new file mode 100644 index 00000000..dda9d587 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf @@ -0,0 +1,22 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + backend "gcs" { + bucket = "UPDATE_PROJECTS_BACKEND" + prefix = "terraform/projects/ml_business_unit/nonproduction" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf.cloud.example b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf.cloud.example new file mode 100644 index 00000000..33cd9944 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/backend.tf.cloud.example @@ -0,0 +1,23 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + cloud { + workspaces { + name = "4-ml-nonproduction" + } + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/common.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/common.auto.tfvars new file mode 120000 index 00000000..39aaa462 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/common.auto.tfvars @@ -0,0 +1 @@ +../../common.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/locals.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/locals.tf new file mode 100644 index 00000000..b10bf71e --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/locals.tf @@ -0,0 +1,19 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +locals { + repo_name = "ml-composer" + business_code = "ml" + business_unit = "ml_business_unit" +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/main.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/main.tf new file mode 100644 index 00000000..ec57d449 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/main.tf @@ -0,0 +1,34 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "bu_folder" { + source = "../../modules/env_folders" + business_code = local.business_code + remote_state_bucket = var.remote_state_bucket + env = var.env +} + +module "ml_env" { + source = "../../modules/ml_env" + + env = var.env + business_code = local.business_code + business_unit = local.business_unit + remote_state_bucket = var.remote_state_bucket + location_gcs = var.location_gcs + tfc_org_name = var.tfc_org_name + business_unit_folder = module.bu_folder.business_unit_folder +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/non-production.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/non-production.auto.tfvars new file mode 120000 index 00000000..e878a833 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/non-production.auto.tfvars @@ -0,0 +1 @@ +../../non-production.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf new file mode 100644 index 00000000..abc30061 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf @@ -0,0 +1,36 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "machine_learning_project_id" { + description = "Project machine learning project." + value = module.ml_env.machine_learning_project_id +} + +output "machine_learning_project_number" { + description = "Project number of machine learning project." + value = module.ml_env.machine_learning_project_number +} + +output "machine_learning_kms_keys" { + description = "Key ID for the machine learning project." + value = module.ml_env.machine_learning_kms_keys +} + + +output "enable_cloudbuild_deploy" { + description = "Enable infra deployment using Cloud Build." + value = local.enable_cloudbuild_deploy +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf.backup b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf.backup new file mode 100644 index 00000000..0159f645 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/outputs.tf.backup @@ -0,0 +1,125 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "base_shared_vpc_project" { + description = "Project sample base project." + value = module.env.base_shared_vpc_project +} + +output "base_shared_vpc_project_sa" { + description = "Project sample base project SA." + value = module.env.base_shared_vpc_project_sa +} + +output "base_subnets_self_links" { + value = module.env.base_subnets_self_links + description = "The self-links of subnets from base environment." +} + +output "floating_project" { + description = "Project sample floating project." + value = module.env.floating_project +} + +output "peering_project" { + description = "Project sample peering project id." + value = module.env.peering_project +} + +output "peering_network" { + description = "Peer network peering resource." + value = module.env.peering_network +} + +output "restricted_shared_vpc_project" { + description = "Project sample restricted project id." + value = module.env.restricted_shared_vpc_project +} + +output "restricted_shared_vpc_project_number" { + description = "Project sample restricted project." + value = module.env.restricted_shared_vpc_project_number +} + +output "restricted_subnets_self_links" { + value = module.env.restricted_subnets_self_links + description = "The self-links of subnets from restricted environment." +} + +output "vpc_service_control_perimeter_name" { + description = "VPC Service Control name." + value = module.env.vpc_service_control_perimeter_name +} + +output "restricted_enabled_apis" { + description = "Activated APIs." + value = module.env.restricted_enabled_apis +} + +output "access_context_manager_policy_id" { + description = "Access Context Manager Policy ID." + value = module.env.access_context_manager_policy_id +} + +output "peering_complete" { + description = "Output to be used as a module dependency." + value = module.env.peering_complete +} + +output "env_kms_project" { + description = "Project sample for KMS usage project ID." + value = module.env.env_kms_project +} + +output "keyring" { + description = "The name of the keyring." + value = module.env.keyring +} + +output "keys" { + description = "List of created key names." + value = module.env.keys +} + +output "bucket" { + description = "The created storage bucket." + value = module.env.bucket +} + +output "peering_subnetwork_self_link" { + description = "The subnetwork self link of the peering network." + value = module.env.peering_subnetwork_self_link +} + +output "iap_firewall_tags" { + description = "The security tags created for IAP (SSH and RDP) firewall rules and to be used on the VM created on step 5-app-infra on the peering network project." + value = module.env.iap_firewall_tags +} + +output "machine_learning_project_id" { + description = "The project ID of the machine learning project." + value = module.ml_env.machine_learning_project +} + +output "machine_learning_project_number" { + description = "The project Number of the machine learning project." + value = module.ml_env.machine_learning_project_number +} + +output "machine_learning_key_id" { + description = "The kms key ID of the machine learning project." + value = module.ml_env.machine_learning_key_id +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/remote.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/remote.tf new file mode 100644 index 00000000..19fd1ffb --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/remote.tf @@ -0,0 +1,60 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.terraform_remote_state.bootstrap.outputs.common_config.org_id + parent_folder = data.terraform_remote_state.bootstrap.outputs.common_config.parent_folder + parent = data.terraform_remote_state.bootstrap.outputs.common_config.parent_id + location_gcs = try(data.terraform_remote_state.bootstrap.outputs.common_config.default_region, var.location_gcs) + billing_account = data.terraform_remote_state.bootstrap.outputs.common_config.billing_account + common_folder_name = data.terraform_remote_state.org.outputs.common_folder_name + common_kms_project_id = data.terraform_remote_state.org.outputs.org_kms_project_id + default_region = data.terraform_remote_state.bootstrap.outputs.common_config.default_region + project_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.project_prefix + folder_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.folder_prefix + projects_remote_bucket_tfstate = data.terraform_remote_state.bootstrap.outputs.projects_gcs_bucket_tfstate + cloud_build_private_worker_pool_id = try(data.terraform_remote_state.bootstrap.outputs.cloud_build_private_worker_pool_id, "") + cloud_builder_artifact_repo = try(data.terraform_remote_state.bootstrap.outputs.cloud_builder_artifact_repo, "") + enable_cloudbuild_deploy = local.cloud_builder_artifact_repo != "" + environment_kms_key_ring = data.terraform_remote_state.environments_env.outputs.key_rings +} + +data "terraform_remote_state" "bootstrap" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/bootstrap/state" + } +} + +data "terraform_remote_state" "org" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/org/state" + } +} + +data "terraform_remote_state" "environments_env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/variables.tf b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/variables.tf new file mode 100644 index 00000000..59a87d1f --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/nonproduction/variables.tf @@ -0,0 +1,78 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "env" { + description = "The environment this deployment belongs to (ie. development)" + type = string +} +variable "default_region" { + description = "Default region to create resources where applicable." + type = string + default = "us-central1" +} + +variable "remote_state_bucket" { + description = "Backend bucket to load Terraform Remote State Data from previous steps." + type = string +} + +variable "location_kms" { + description = "Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket)" + type = string + default = "us" +} + +variable "location_gcs" { + description = "Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring)" + type = string + default = "US" +} + +variable "peering_module_depends_on" { + description = "List of modules or resources peering module depends on." + type = list(any) + default = [] +} + +variable "tfc_org_name" { + description = "Name of the TFC organization." + type = string + default = "" +} + +variable "project_budget" { + description = < +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| default\_region | Default region to create resources where applicable. | `string` | `"us-central1"` | no | +| env | The environment this deployment belongs to (ie. development) | `string` | n/a | yes | +| instance\_region | Region which the peered subnet will be created (Should be same region as the VM that will be created on step 5-app-infra on the peering project). | `string` | `"us-central1"` | no | +| location\_gcs | Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring) | `string` | `"US"` | no | +| location\_kms | Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket) | `string` | `"us"` | no | +| peering\_module\_depends\_on | List of modules or resources peering module depends on. | `list(any)` | `[]` | no | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | +| tfc\_org\_name | Name of the TFC organization | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| enable\_cloudbuild\_deploy | Enable infra deployment using Cloud Build. | +| machine\_learning\_kms\_keys | Key ID for the machine learning project. | +| machine\_learning\_project\_id | Project machine learning project. | +| machine\_learning\_project\_number | Project number of machine learning project. | + + diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf new file mode 100644 index 00000000..f900eb0c --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf @@ -0,0 +1,22 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + backend "gcs" { + bucket = "UPDATE_PROJECTS_BACKEND" + prefix = "terraform/projects/ml_business_unit/production" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf.cloud.example b/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf.cloud.example new file mode 100644 index 00000000..704da6c2 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/backend.tf.cloud.example @@ -0,0 +1,23 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + cloud { + workspaces { + name = "4-ml-production" + } + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/common.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/production/common.auto.tfvars new file mode 120000 index 00000000..39aaa462 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/common.auto.tfvars @@ -0,0 +1 @@ +../../common.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/locals.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/locals.tf new file mode 100644 index 00000000..b10bf71e --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/locals.tf @@ -0,0 +1,19 @@ +# Copyright 2024 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +locals { + repo_name = "ml-composer" + business_code = "ml" + business_unit = "ml_business_unit" +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/main.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/main.tf new file mode 100644 index 00000000..ec57d449 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/main.tf @@ -0,0 +1,34 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "bu_folder" { + source = "../../modules/env_folders" + business_code = local.business_code + remote_state_bucket = var.remote_state_bucket + env = var.env +} + +module "ml_env" { + source = "../../modules/ml_env" + + env = var.env + business_code = local.business_code + business_unit = local.business_unit + remote_state_bucket = var.remote_state_bucket + location_gcs = var.location_gcs + tfc_org_name = var.tfc_org_name + business_unit_folder = module.bu_folder.business_unit_folder +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf new file mode 100644 index 00000000..947cd723 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf @@ -0,0 +1,35 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "machine_learning_project_id" { + description = "Project machine learning project." + value = module.ml_env.machine_learning_project_id +} + +output "machine_learning_project_number" { + description = "Project number of machine learning project." + value = module.ml_env.machine_learning_project_number +} + +output "machine_learning_kms_keys" { + description = "Key ID for the machine learning project." + value = module.ml_env.machine_learning_kms_keys +} + +output "enable_cloudbuild_deploy" { + description = "Enable infra deployment using Cloud Build." + value = local.enable_cloudbuild_deploy +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf.backup b/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf.backup new file mode 100644 index 00000000..0159f645 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/outputs.tf.backup @@ -0,0 +1,125 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "base_shared_vpc_project" { + description = "Project sample base project." + value = module.env.base_shared_vpc_project +} + +output "base_shared_vpc_project_sa" { + description = "Project sample base project SA." + value = module.env.base_shared_vpc_project_sa +} + +output "base_subnets_self_links" { + value = module.env.base_subnets_self_links + description = "The self-links of subnets from base environment." +} + +output "floating_project" { + description = "Project sample floating project." + value = module.env.floating_project +} + +output "peering_project" { + description = "Project sample peering project id." + value = module.env.peering_project +} + +output "peering_network" { + description = "Peer network peering resource." + value = module.env.peering_network +} + +output "restricted_shared_vpc_project" { + description = "Project sample restricted project id." + value = module.env.restricted_shared_vpc_project +} + +output "restricted_shared_vpc_project_number" { + description = "Project sample restricted project." + value = module.env.restricted_shared_vpc_project_number +} + +output "restricted_subnets_self_links" { + value = module.env.restricted_subnets_self_links + description = "The self-links of subnets from restricted environment." +} + +output "vpc_service_control_perimeter_name" { + description = "VPC Service Control name." + value = module.env.vpc_service_control_perimeter_name +} + +output "restricted_enabled_apis" { + description = "Activated APIs." + value = module.env.restricted_enabled_apis +} + +output "access_context_manager_policy_id" { + description = "Access Context Manager Policy ID." + value = module.env.access_context_manager_policy_id +} + +output "peering_complete" { + description = "Output to be used as a module dependency." + value = module.env.peering_complete +} + +output "env_kms_project" { + description = "Project sample for KMS usage project ID." + value = module.env.env_kms_project +} + +output "keyring" { + description = "The name of the keyring." + value = module.env.keyring +} + +output "keys" { + description = "List of created key names." + value = module.env.keys +} + +output "bucket" { + description = "The created storage bucket." + value = module.env.bucket +} + +output "peering_subnetwork_self_link" { + description = "The subnetwork self link of the peering network." + value = module.env.peering_subnetwork_self_link +} + +output "iap_firewall_tags" { + description = "The security tags created for IAP (SSH and RDP) firewall rules and to be used on the VM created on step 5-app-infra on the peering network project." + value = module.env.iap_firewall_tags +} + +output "machine_learning_project_id" { + description = "The project ID of the machine learning project." + value = module.ml_env.machine_learning_project +} + +output "machine_learning_project_number" { + description = "The project Number of the machine learning project." + value = module.ml_env.machine_learning_project_number +} + +output "machine_learning_key_id" { + description = "The kms key ID of the machine learning project." + value = module.ml_env.machine_learning_key_id +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/production.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/production/production.auto.tfvars new file mode 120000 index 00000000..be31a2ed --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/production.auto.tfvars @@ -0,0 +1 @@ +../../production.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/remote.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/remote.tf new file mode 100644 index 00000000..19fd1ffb --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/remote.tf @@ -0,0 +1,60 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.terraform_remote_state.bootstrap.outputs.common_config.org_id + parent_folder = data.terraform_remote_state.bootstrap.outputs.common_config.parent_folder + parent = data.terraform_remote_state.bootstrap.outputs.common_config.parent_id + location_gcs = try(data.terraform_remote_state.bootstrap.outputs.common_config.default_region, var.location_gcs) + billing_account = data.terraform_remote_state.bootstrap.outputs.common_config.billing_account + common_folder_name = data.terraform_remote_state.org.outputs.common_folder_name + common_kms_project_id = data.terraform_remote_state.org.outputs.org_kms_project_id + default_region = data.terraform_remote_state.bootstrap.outputs.common_config.default_region + project_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.project_prefix + folder_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.folder_prefix + projects_remote_bucket_tfstate = data.terraform_remote_state.bootstrap.outputs.projects_gcs_bucket_tfstate + cloud_build_private_worker_pool_id = try(data.terraform_remote_state.bootstrap.outputs.cloud_build_private_worker_pool_id, "") + cloud_builder_artifact_repo = try(data.terraform_remote_state.bootstrap.outputs.cloud_builder_artifact_repo, "") + enable_cloudbuild_deploy = local.cloud_builder_artifact_repo != "" + environment_kms_key_ring = data.terraform_remote_state.environments_env.outputs.key_rings +} + +data "terraform_remote_state" "bootstrap" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/bootstrap/state" + } +} + +data "terraform_remote_state" "org" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/org/state" + } +} + +data "terraform_remote_state" "environments_env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/production/variables.tf b/docs/assets/terraform/4-projects/ml_business_unit/production/variables.tf new file mode 100644 index 00000000..ac47f95a --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/production/variables.tf @@ -0,0 +1,59 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "env" { + description = "The environment this deployment belongs to (ie. development)" + type = string +} +variable "default_region" { + description = "Default region to create resources where applicable." + type = string + default = "us-central1" +} +variable "remote_state_bucket" { + description = "Backend bucket to load Terraform Remote State Data from previous steps." + type = string +} + +variable "location_kms" { + description = "Case-Sensitive Location for KMS Keyring (Should be same region as the GCS Bucket)" + type = string + default = "us" +} + +variable "location_gcs" { + description = "Case-Sensitive Location for GCS Bucket (Should be same region as the KMS Keyring)" + type = string + default = "US" +} + +variable "peering_module_depends_on" { + description = "List of modules or resources peering module depends on." + type = list(any) + default = [] +} + +variable "tfc_org_name" { + description = "Name of the TFC organization" + type = string + default = "" +} + +variable "instance_region" { + description = "Region which the peered subnet will be created (Should be same region as the VM that will be created on step 5-app-infra on the peering project)." + type = string + default = "us-central1" +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/README.md b/docs/assets/terraform/4-projects/ml_business_unit/shared/README.md new file mode 100644 index 00000000..b65cb8a9 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/README.md @@ -0,0 +1,40 @@ + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| cloud\_source\_artifacts\_repo\_name | Name to give the could source repository for Artifacts | `string` | n/a | yes | +| cloud\_source\_service\_catalog\_repo\_name | Name to give the cloud source repository for Service Catalog | `string` | n/a | yes | +| default\_region | Default region to create resources where applicable. | `string` | `"us-central1"` | no | +| gcs\_bucket\_prefix | Name prefix to be used for GCS Bucket | `string` | `"bkt"` | no | +| key\_rotation\_period | Rotation period in seconds to be used for KMS Key | `string` | `"7776000s"` | no | +| keyring\_name | Name to be used for KMS Keyring | `string` | `"sample-keyring"` | no | +| location\_gcs | Case-Sensitive Location for GCS Bucket | `string` | `"US"` | no | +| location\_kms | Case-Sensitive Location for KMS Keyring | `string` | `"us"` | no | +| prevent\_destroy | Prevent Project Key destruction. | `bool` | `true` | no | +| project\_budget | Budget configuration.
budget\_amount: The amount to use as the budget.
alert\_spent\_percents: A list of percentages of the budget to alert on when threshold is exceeded.
alert\_pubsub\_topic: The name of the Cloud Pub/Sub topic where budget related messages will be published, in the form of `projects/{project_id}/topics/{topic_id}`.
alert\_spend\_basis: The type of basis used to determine if spend has passed the threshold. Possible choices are `CURRENT_SPEND` or `FORECASTED_SPEND` (default). | 
object({
    budget_amount        = optional(number, 1000)
    alert_spent_percents = optional(list(number), [1.2])
    alert_pubsub_topic   = optional(string, null)
    alert_spend_basis    = optional(string, "FORECASTED_SPEND")
  })
| `{}` | no | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | +| tfc\_org\_name | Name of the TFC organization | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| apply\_triggers\_id | CB apply triggers | +| artifact\_buckets | GCS Buckets to store Cloud Build Artifacts | +| artifacts\_repo\_id | ID of the Artifacts repository | +| artifacts\_repo\_name | The name of the Artifacts repository | +| cloudbuild\_project\_id | n/a | +| common\_artifacts\_project\_id | App Infra Artifacts Project ID | +| default\_region | Default region to create resources where applicable. | +| enable\_cloudbuild\_deploy | Enable infra deployment using Cloud Build. | +| log\_buckets | GCS Buckets to store Cloud Build logs | +| plan\_triggers\_id | CB plan triggers | +| repos | CSRs to store source code | +| service\_catalog\_project\_id | Service Catalog Project ID. | +| service\_catalog\_repo\_id | ID of the Service Catalog repository | +| service\_catalog\_repo\_name | The name of the Service Catalog repository | +| state\_buckets | GCS Buckets to store TF state | +| terraform\_service\_accounts | APP Infra Pipeline Terraform Accounts. | + + diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf new file mode 100644 index 00000000..130bb23c --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf @@ -0,0 +1,22 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + backend "gcs" { + bucket = "UPDATE_PROJECTS_BACKEND" + prefix = "terraform/projects/ml_business_unit/shared" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf.cloud.example b/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf.cloud.example new file mode 100644 index 00000000..5dfb05be --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/backend.tf.cloud.example @@ -0,0 +1,23 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + cloud { + workspaces { + name = "4-bu2-shared" + } + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/common.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/shared/common.auto.tfvars new file mode 120000 index 00000000..39aaa462 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/common.auto.tfvars @@ -0,0 +1 @@ +../../common.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/example_infra_pipeline.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/example_infra_pipeline.tf new file mode 100644 index 00000000..e12c51eb --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/example_infra_pipeline.tf @@ -0,0 +1,90 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + repo_names = [ + "ml-artifact-publish", + "ml-service-catalog", + "ml-machine-learning", + ] +} + +module "app_infra_cloudbuild_project" { + source = "../../modules/ml_single_project" + count = local.enable_cloudbuild_deploy ? 1 : 0 + + org_id = local.org_id + billing_account = local.billing_account + folder_id = local.common_folder_name + environment = "common" + project_budget = var.project_budget + project_prefix = local.project_prefix + key_rings = local.shared_kms_key_ring + remote_state_bucket = var.remote_state_bucket + activate_apis = [ + "cloudbuild.googleapis.com", + "sourcerepo.googleapis.com", + "cloudkms.googleapis.com", + "iam.googleapis.com", + "artifactregistry.googleapis.com", + "cloudresourcemanager.googleapis.com", + "serviceusage.googleapis.com", + "bigquery.googleapis.com", + ] + # Metadata + project_suffix = "infra-pipeline" + application_name = "app-infra-pipelines" + billing_code = "1234" + primary_contact = "example@example.com" + secondary_contact = "example2@example.com" + business_code = "ml" + + environment_kms_project_id = "" + project_name = "prj-c-ml-infra-pipeline" + prevent_destroy = var.prevent_destroy +} + +module "infra_pipelines" { + source = "../../modules/infra_pipelines" + count = local.enable_cloudbuild_deploy ? 1 : 0 + + org_id = local.org_id + cloudbuild_project_id = module.app_infra_cloudbuild_project[0].project_id + cloud_builder_artifact_repo = local.cloud_builder_artifact_repo + remote_tfstate_bucket = local.projects_remote_bucket_tfstate + billing_account = local.billing_account + default_region = var.default_region + app_infra_repos = local.repo_names + private_worker_pool_id = local.cloud_build_private_worker_pool_id +} + +resource "google_kms_key_ring_iam_member" "key_ring" { + for_each = { for k in flatten([for kms in local.shared_kms_key_ring : [for name, email in module.infra_pipelines[0].terraform_service_accounts : { key = "${kms}--${name}", kms = kms, email = email }]]) : k.key => k } + key_ring_id = each.value.kms + role = "roles/cloudkms.admin" + member = "serviceAccount:${each.value.email}" +} + +/** + * When Jenkins CICD is used for deployment this resource + * is created to terraform validation works. + * Without this resource, this module creates zero resources + * and it breaks terraform validation throwing the error below: + * ERROR: [Terraform plan json does not contain resource_changes key] + */ +resource "null_resource" "jenkins_cicd" { + count = !local.enable_cloudbuild_deploy ? 1 : 0 +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/ml_infra_projects.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/ml_infra_projects.tf new file mode 100644 index 00000000..7640d30e --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/ml_infra_projects.tf @@ -0,0 +1,36 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +module "ml_infra_projects" { + source = "../../modules/ml_infra_projects" + + org_id = local.org_id + folder_id = local.common_folder_name + billing_account = local.billing_account + environment = "common" + key_rings = local.shared_kms_key_ring + business_code = "ml" + billing_code = "1234" + primary_contact = "example@example.com" + secondary_contact = "example2@example.com" + cloud_source_artifacts_repo_name = var.cloud_source_artifacts_repo_name + cloud_source_service_catalog_repo_name = var.cloud_source_service_catalog_repo_name + remote_state_bucket = var.remote_state_bucket + artifacts_infra_pipeline_sa = module.infra_pipelines[0].terraform_service_accounts["ml-artifact-publish"] + service_catalog_infra_pipeline_sa = module.infra_pipelines[0].terraform_service_accounts["ml-service-catalog"] + environment_kms_project_id = "" + prevent_destroy = var.prevent_destroy +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/outputs.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/outputs.tf new file mode 100644 index 00000000..88c5d34c --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/outputs.tf @@ -0,0 +1,94 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "default_region" { + description = "Default region to create resources where applicable." + value = try(module.infra_pipelines[0].default_region, "") +} + +output "cloudbuild_project_id" { + value = try(module.app_infra_cloudbuild_project[0].project_id, "") +} + +output "terraform_service_accounts" { + description = "APP Infra Pipeline Terraform Accounts." + value = try(module.infra_pipelines[0].terraform_service_accounts, {}) +} + +output "repos" { + description = "CSRs to store source code" + value = try(module.infra_pipelines[0].repos, toset([])) +} + +output "artifact_buckets" { + description = "GCS Buckets to store Cloud Build Artifacts" + value = try(module.infra_pipelines[0].artifact_buckets, {}) +} + +output "state_buckets" { + description = "GCS Buckets to store TF state" + value = try(module.infra_pipelines[0].state_buckets, {}) +} + +output "log_buckets" { + description = "GCS Buckets to store Cloud Build logs" + value = try(module.infra_pipelines[0].log_buckets, {}) +} + +output "plan_triggers_id" { + description = "CB plan triggers" + value = try(module.infra_pipelines[0].plan_triggers_id, []) +} + +output "apply_triggers_id" { + description = "CB apply triggers" + value = try(module.infra_pipelines[0].apply_triggers_id, []) +} + +output "enable_cloudbuild_deploy" { + description = "Enable infra deployment using Cloud Build." + value = local.enable_cloudbuild_deploy +} + +output "service_catalog_project_id" { + description = "Service Catalog Project ID." + value = module.ml_infra_projects.service_catalog_project_id +} + +output "common_artifacts_project_id" { + description = "App Infra Artifacts Project ID" + value = module.ml_infra_projects.common_artifacts_project_id +} + +output "service_catalog_repo_name" { + description = "The name of the Service Catalog repository" + value = module.ml_infra_projects.service_catalog_repo_name +} + +output "service_catalog_repo_id" { + description = "ID of the Service Catalog repository" + value = module.ml_infra_projects.service_catalog_repo_id +} + +output "artifacts_repo_name" { + description = "The name of the Artifacts repository" + value = module.ml_infra_projects.artifacts_repo_name +} + +output "artifacts_repo_id" { + description = "ID of the Artifacts repository" + value = module.ml_infra_projects.artifacts_repo_id +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf new file mode 100644 index 00000000..74696cd0 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf @@ -0,0 +1,51 @@ +/** + * Copyright 2022 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.terraform_remote_state.bootstrap.outputs.common_config.org_id + parent_folder = data.terraform_remote_state.bootstrap.outputs.common_config.parent_folder + parent = data.terraform_remote_state.bootstrap.outputs.common_config.parent_id + location_gcs = try(data.terraform_remote_state.bootstrap.outputs.common_config.default_region, var.location_gcs) + billing_account = data.terraform_remote_state.bootstrap.outputs.common_config.billing_account + common_folder_name = data.terraform_remote_state.org.outputs.common_folder_name + common_kms_project_id = data.terraform_remote_state.org.outputs.org_kms_project_id + default_region = data.terraform_remote_state.bootstrap.outputs.common_config.default_region + project_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.project_prefix + folder_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.folder_prefix + projects_remote_bucket_tfstate = data.terraform_remote_state.bootstrap.outputs.projects_gcs_bucket_tfstate + cloud_build_private_worker_pool_id = try(data.terraform_remote_state.bootstrap.outputs.cloud_build_private_worker_pool_id, "") + cloud_builder_artifact_repo = try(data.terraform_remote_state.bootstrap.outputs.cloud_builder_artifact_repo, "") + enable_cloudbuild_deploy = local.cloud_builder_artifact_repo != "" + shared_kms_key_ring = data.terraform_remote_state.org.outputs.key_rings +} + +data "terraform_remote_state" "bootstrap" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/bootstrap/state" + } +} + +data "terraform_remote_state" "org" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/org/state" + } +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf.cloud.example b/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf.cloud.example new file mode 100644 index 00000000..525ba30a --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/remote.tf.cloud.example @@ -0,0 +1,40 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.org_id + parent_folder = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.parent_folder + parent = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.parent_id + billing_account = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.billing_account + common_folder_name = data.tfe_outputs.org.nonsensitive_values.common_folder_name + default_region = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.default_region + project_prefix = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.project_prefix + folder_prefix = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.folder_prefix + projects_remote_bucket_tfstate = "" + cloud_build_private_worker_pool_id = try(data.tfe_outputs.bootstrap.nonsensitive_values.cloud_build_private_worker_pool_id, "") + cloud_builder_artifact_repo = try(data.tfe_outputs.bootstrap.nonsensitive_values.cloud_builder_artifact_repo, "") + enable_cloudbuild_deploy = local.cloud_builder_artifact_repo != "" +} + +data "tfe_outputs" "bootstrap" { + organization = var.tfc_org_name + workspace = "0-shared" +} + +data "tfe_outputs" "org" { + organization = var.tfc_org_name + workspace = "1-shared" +} diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/shared.auto.tfvars b/docs/assets/terraform/4-projects/ml_business_unit/shared/shared.auto.tfvars new file mode 120000 index 00000000..b7f8387a --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/shared.auto.tfvars @@ -0,0 +1 @@ +../../shared.auto.tfvars \ No newline at end of file diff --git a/docs/assets/terraform/4-projects/ml_business_unit/shared/variables.tf b/docs/assets/terraform/4-projects/ml_business_unit/shared/variables.tf new file mode 100644 index 00000000..7efc60d6 --- /dev/null +++ b/docs/assets/terraform/4-projects/ml_business_unit/shared/variables.tf @@ -0,0 +1,95 @@ +/** + * Copyright 2021 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "default_region" { + description = "Default region to create resources where applicable." + type = string + default = "us-central1" +} + +variable "location_gcs" { + description = "Case-Sensitive Location for GCS Bucket" + type = string + default = "US" +} + +variable "location_kms" { + description = "Case-Sensitive Location for KMS Keyring" + type = string + default = "us" +} + +variable "keyring_name" { + description = "Name to be used for KMS Keyring" + type = string + default = "sample-keyring" +} + +variable "gcs_bucket_prefix" { + description = "Name prefix to be used for GCS Bucket" + type = string + default = "bkt" +} + +variable "project_budget" { + description = < nr } + + subnetwork = each.value.subnet + role = "roles/compute.networkUser" + region = each.value.region + project = local.shared_vpc_host_project_id + member = "serviceAccount:${google_project_service_identity.notebooks.email}" +} + +resource "google_kms_crypto_key_iam_member" "secrets" { + for_each = module.machine_learning_project.kms_keys + + crypto_key_id = each.value.id + role = "roles/cloudkms.cryptoKeyEncrypterDecrypter" + member = "serviceAccount:${google_project_service_identity.secrets.email}" + + depends_on = [time_sleep.wait_30_seconds] +} + +// Allow machine-learning sa access to service catalog cloud source repository +resource "google_sourcerepo_repository_iam_member" "read" { + project = local.service_catalog_project_id + repository = local.service_catalog_repo_name + role = "roles/viewer" + member = "serviceAccount:${local.app_infra_pipeline_service_accounts["ml-machine-learning"]}" +} + +// Add Browser Role to CloudBuild at Env Folder + +resource "google_folder_iam_member" "name" { + folder = local.env_folder_name + role = "roles/browser" + member = "serviceAccount:${google_project_service_identity.cloud_build.email}" +} + +// Add Artifact Registry Access to Vertex AI Agent + +resource "google_project_iam_member" "access_artifacts" { + count = var.env == "production" ? 1 : 0 + project = local.common_artifact_project_id + role = "roles/artifactregistry.reader" + member = "serviceAccount:service-${module.machine_learning_project.project_number}@gcp-sa-aiplatform.iam.gserviceaccount.com" + + depends_on = [time_sleep.wait_30_seconds] +} diff --git a/docs/assets/terraform/4-projects/modules/ml_env/outputs.tf b/docs/assets/terraform/4-projects/modules/ml_env/outputs.tf new file mode 100644 index 00000000..6c5085b2 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/outputs.tf @@ -0,0 +1,30 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "machine_learning_project_id" { + description = "Project machine learning project." + value = module.machine_learning_project.project_id +} + +output "machine_learning_project_number" { + description = "Project number of machine learning project." + value = module.machine_learning_project.project_number +} + +output "machine_learning_kms_keys" { + description = "Key ID for the machine learning project." + value = module.machine_learning_project.kms_keys +} diff --git a/docs/assets/terraform/4-projects/modules/ml_env/pubsub.tf b/docs/assets/terraform/4-projects/modules/ml_env/pubsub.tf new file mode 100644 index 00000000..1a849f24 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/pubsub.tf @@ -0,0 +1,34 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + pubsub_topic_name = "secret-rotation-notifications" +} + +// Secret rotation notification +resource "google_pubsub_topic" "secret_rotations" { + name = local.pubsub_topic_name + project = module.machine_learning_project.project_id +} + +resource "google_pubsub_topic_iam_member" "pubsub_binding" { + topic = google_pubsub_topic.secret_rotations.name + project = module.machine_learning_project.project_id + role = "roles/pubsub.publisher" + member = "serviceAccount:service-${module.machine_learning_project.project_number}@gcp-sa-secretmanager.iam.gserviceaccount.com" + + depends_on = [time_sleep.wait_30_seconds] +} diff --git a/docs/assets/terraform/4-projects/modules/ml_env/remote.tf b/docs/assets/terraform/4-projects/modules/ml_env/remote.tf new file mode 100644 index 00000000..085940e9 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/remote.tf @@ -0,0 +1,91 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.terraform_remote_state.bootstrap.outputs.common_config.org_id + billing_account = data.terraform_remote_state.bootstrap.outputs.common_config.billing_account + project_prefix = data.terraform_remote_state.bootstrap.outputs.common_config.project_prefix + projects_backend_bucket = data.terraform_remote_state.bootstrap.outputs.projects_gcs_bucket_tfstate + perimeter_name = data.terraform_remote_state.network_env.outputs.restricted_service_perimeter_name + base_network_self_link = data.terraform_remote_state.network_env.outputs.base_network_self_link + base_subnets_self_links = data.terraform_remote_state.network_env.outputs.base_subnets_self_links + base_host_project_id = data.terraform_remote_state.network_env.outputs.base_host_project_id + restricted_host_project_id = data.terraform_remote_state.network_env.outputs.restricted_host_project_id + restricted_subnets_self_links = data.terraform_remote_state.network_env.outputs.restricted_subnets_self_links + access_context_manager_policy_id = data.terraform_remote_state.network_env.outputs.access_context_manager_policy_id + env_folder_name = data.terraform_remote_state.environments_env.outputs.env_folder + app_infra_pipeline_service_accounts = data.terraform_remote_state.business_unit_shared.outputs.terraform_service_accounts + enable_cloudbuild_deploy = data.terraform_remote_state.business_unit_shared.outputs.enable_cloudbuild_deploy + environments_kms_key_ring = data.terraform_remote_state.environments_env.outputs.key_rings + environment_kms_project_id = data.terraform_remote_state.environments_env.outputs.env_kms_project_id + default_region = data.terraform_remote_state.business_unit_shared.outputs.default_region + service_catalog_project_id = data.terraform_remote_state.business_unit_shared.outputs.service_catalog_project_id + service_catalog_repo_name = data.terraform_remote_state.business_unit_shared.outputs.service_catalog_repo_name + common_artifact_project_id = data.terraform_remote_state.business_unit_shared.outputs.common_artifacts_project_id + + restricted_subnets_region = flatten([ + for subnet in local.restricted_subnets_self_links : + { + subnet = element(split("/", subnet), index(split("/", subnet), "subnetworks", ) + 1, ) + region = element(split("/", subnet), index(split("/", subnet), "regions") + 1, ) + } + ]) +} + +data "terraform_remote_state" "bootstrap" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/bootstrap/state" + } +} + +data "terraform_remote_state" "org" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/org/state" + } +} + +data "terraform_remote_state" "network_env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/networks/${var.env}" + } +} + +data "terraform_remote_state" "environments_env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } +} + +data "terraform_remote_state" "business_unit_shared" { + backend = "gcs" + + config = { + bucket = local.projects_backend_bucket + prefix = "terraform/projects/${var.business_unit}/shared" + } +} diff --git a/docs/assets/terraform/4-projects/modules/ml_env/remote.tf.cloud.example b/docs/assets/terraform/4-projects/modules/ml_env/remote.tf.cloud.example new file mode 100644 index 00000000..a89c93c5 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/remote.tf.cloud.example @@ -0,0 +1,52 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + org_id = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.org_id + billing_account = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.billing_account + project_prefix = data.tfe_outputs.bootstrap.nonsensitive_values.common_config.project_prefix + projects_backend_bucket = "" + perimeter_name = data.tfe_outputs.network_env.nonsensitive_values.restricted_service_perimeter_name + base_network_self_link = data.tfe_outputs.network_env.nonsensitive_values.base_network_self_link + base_subnets_self_links = data.tfe_outputs.network_env.nonsensitive_values.base_subnets_self_links + base_host_project_id = data.tfe_outputs.network_env.nonsensitive_values.base_host_project_id + restricted_host_project_id = data.tfe_outputs.network_env.nonsensitive_values.restricted_host_project_id + restricted_subnets_self_links = data.tfe_outputs.network_env.nonsensitive_values.restricted_subnets_self_links + access_context_manager_policy_id = data.tfe_outputs.network_env.nonsensitive_values.access_context_manager_policy_id + env_folder_name = data.tfe_outputs.environments_env.nonsensitive_values.env_folder + app_infra_pipeline_service_accounts = data.tfe_outputs.business_unit_shared.nonsensitive_values.terraform_service_accounts + enable_cloudbuild_deploy = data.tfe_outputs.business_unit_shared.nonsensitive_values.enable_cloudbuild_deploy +} + +data "tfe_outputs" "bootstrap" { + organization = var.tfc_org_name + workspace = "0-shared" +} + +data "tfe_outputs" "org" { + organization = var.tfc_org_name + workspace = "1-shared" +} + +data "tfe_outputs" "environments_env" { + organization = var.tfc_org_name + workspace = "2-${var.env}" +} + +data "tfe_outputs" "business_unit_shared" { + organization = var.tfc_org_name + workspace = "4-${var.business_code}-shared" +} diff --git a/docs/assets/terraform/4-projects/modules/ml_env/secret.tf b/docs/assets/terraform/4-projects/modules/ml_env/secret.tf new file mode 100644 index 00000000..10a7477e --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/secret.tf @@ -0,0 +1,62 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + secrets = [ + "github-api-token", + ] +} + +resource "google_secret_manager_secret" "secret" { + for_each = toset(local.secrets) + project = module.machine_learning_project.project_id + secret_id = each.key + + #Set up Automatic Rotation of Secrets + #Control ID: SM-CO-6.2 + #NIST 800-53: SC-12 SC-13 + + rotation { + next_rotation_time = formatdate("YYYY-MM-DD'T'hh:mm:ss'Z'", timeadd(timestamp(), "720h")) + rotation_period = "43200s" + } + + topics { + name = google_pubsub_topic.secret_rotations.id + } + + #Automatic Secret Replication + #Control ID: SM-CO-6.1 + #NIST 800-53: SC-12 SC-13 + replication { + user_managed { + replicas { + location = var.region + + #Customer Managed Encryption Keys + #Control ID: COM-CO-2.3 + #NIST 800-53: SC-12 SC-13 + #CRI Profile: PR.DS-1.1 PR.DS-1.2 PR.DS-2.1 PR.DS-2.2 PR.DS-5.1 + customer_managed_encryption { + kms_key_name = module.machine_learning_project.kms_keys[var.region].id + } + } + } + } + + depends_on = [google_kms_crypto_key_iam_member.secrets, google_pubsub_topic_iam_member.pubsub_binding] +} + diff --git a/docs/assets/terraform/4-projects/modules/ml_env/variables.tf b/docs/assets/terraform/4-projects/modules/ml_env/variables.tf new file mode 100644 index 00000000..592d10fc --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_env/variables.tf @@ -0,0 +1,101 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "business_code" { + description = "The business code (ex. bu1)." + type = string +} + +variable "business_unit" { + description = "The business (ex. business_unit_1)." + type = string +} + +variable "env" { + description = "The environment to prepare (ex. development)." + type = string +} + +variable "business_unit_folder" { + description = "The Business Unit Folder (Created in base_env)" + type = string +} + +variable "project_budget" { + description = < v } : { for k, v in google_kms_crypto_key.ephemeral_kms_keys : split("/", k)[3] => v } +} + +resource "google_kms_crypto_key" "ephemeral_kms_keys" { + for_each = toset(local.ephemeral_keys_for_each) + + name = var.project_name + key_ring = each.key + rotation_period = var.key_rotation_period + lifecycle { + prevent_destroy = false + } +} + +resource "google_kms_crypto_key" "kms_keys" { + for_each = toset(local.keys_for_each) + + name = var.project_name + key_ring = each.key + rotation_period = var.key_rotation_period + lifecycle { + prevent_destroy = true + } +} diff --git a/docs/assets/terraform/4-projects/modules/ml_kms_key/outputs.tf b/docs/assets/terraform/4-projects/modules/ml_kms_key/outputs.tf new file mode 100644 index 00000000..b456923a --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_kms_key/outputs.tf @@ -0,0 +1,20 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "kms_keys" { + description = "Keys created for the project." + value = local.output_keys +} diff --git a/docs/assets/terraform/4-projects/modules/ml_kms_key/variables.tf b/docs/assets/terraform/4-projects/modules/ml_kms_key/variables.tf new file mode 100644 index 00000000..c17ee08e --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_kms_key/variables.tf @@ -0,0 +1,36 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "key_rings" { + description = "Keyrings to attach project key to." + type = list(string) +} + +variable "project_name" { + description = "Project Name." + type = string +} + +variable "key_rotation_period" { + description = "Rotation period in seconds to be used for KMS Key." + type = string + default = "7776000s" +} + +variable "prevent_destroy" { + description = "Prevent Key destruction." + type = bool +} diff --git a/docs/assets/terraform/4-projects/modules/ml_kms_key/versions.tf b/docs/assets/terraform/4-projects/modules/ml_kms_key/versions.tf new file mode 100644 index 00000000..2ff08fd4 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_kms_key/versions.tf @@ -0,0 +1,19 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + required_version = ">= 1.3" +} diff --git a/docs/assets/terraform/4-projects/modules/ml_single_project/README.md b/docs/assets/terraform/4-projects/modules/ml_single_project/README.md new file mode 100644 index 00000000..8165effa --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_single_project/README.md @@ -0,0 +1,52 @@ +# Machine Learning Single Project + +Create and manage a Google Cloud project with various configurations and roles required for application infrastructure and pipeline service accounts. It includes the setup of IAM roles, VPC networking, KMS keys, and budget alerts. The module leverages the terraform-google-modules/project-factory/google module for project creation and management. + + + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| activate\_apis | The api to activate for the GCP project. | `list(string)` | `[]` | no | +| app\_infra\_pipeline\_service\_accounts | The Service Accounts from App Infra Pipeline. | `map(string)` | `{}` | no | +| application\_name | The name of application where GCP resources relate. | `string` | n/a | yes | +| billing\_account | The ID of the billing account to associated this project with. | `string` | n/a | yes | +| billing\_code | The code that's used to provide chargeback information. | `string` | n/a | yes | +| business\_code | The code that describes which business unit owns the project. | `string` | `"abcd"` | no | +| default\_service\_account | Project default service account setting: can be one of `delete`, `depriviledge`, `keep` or `disable`. | `string` | `"disable"` | no | +| enable\_cloudbuild\_deploy | Enable infra deployment using Cloud Build. | `bool` | `false` | no | +| environment | The environment the project belongs to. | `string` | n/a | yes | +| environment\_kms\_project\_id | Environment level KMS Project ID. | `string` | n/a | yes | +| folder\_id | The folder id where project will be created. | `string` | n/a | yes | +| key\_rings | Keyrings to attach project key to. | `list(string)` | n/a | yes | +| key\_rotation\_period | Rotation period in seconds to be used for KMS Key. | `string` | `"7776000s"` | no | +| org\_id | The Organization ID. | `string` | n/a | yes | +| prevent\_destroy | Prevent Key destruction. | `bool` | n/a | yes | +| primary\_contact | The primary email contact for the project. | `string` | n/a | yes | +| project\_budget | Budget configuration.
budget\_amount: The amount to use as the budget.
alert\_spent\_percents: A list of percentages of the budget to alert on when threshold is exceeded.
alert\_pubsub\_topic: The name of the Cloud Pub/Sub topic where budget related messages will be published, in the form of `projects/{project_id}/topics/{topic_id}`.
alert\_spend\_basis: The type of basis used to determine if spend has passed the threshold. Possible choices are `CURRENT_SPEND` or `FORECASTED_SPEND` (default). | 
object({
    budget_amount        = optional(number, 1000)
    alert_spent_percents = optional(list(number), [1.2])
    alert_pubsub_topic   = optional(string, null)
    alert_spend_basis    = optional(string, "FORECASTED_SPEND")
  })
| `{}` | no | +| project\_name | Project Name. | `string` | n/a | yes | +| project\_prefix | Name prefix to use for projects created. | `string` | `"prj"` | no | +| project\_suffix | The name of the GCP project. Max 16 characters with 3 character business unit code. | `string` | n/a | yes | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | +| sa\_roles | A list of roles to give the Service Account from App Infra Pipeline. | `map(list(string))` | `{}` | no | +| secondary\_contact | The secondary email contact for the project. | `string` | `""` | no | +| shared\_vpc\_host\_project\_id | Shared VPC host project ID. | `string` | `""` | no | +| shared\_vpc\_subnets | List of the shared vpc subnets self links. | `list(string)` | `[]` | no | +| vpc\_service\_control\_attach\_enabled | Whether the project will be attached to a VPC Service Control Perimeter. | `bool` | `false` | no | +| vpc\_service\_control\_perimeter\_name | The name of a VPC Service Control Perimeter to add the created project to. | `string` | `null` | no | +| vpc\_service\_control\_sleep\_duration | The duration to sleep in seconds before adding the project to a shared VPC after the project is added to the VPC Service Control Perimeter. | `string` | `"5s"` | no | +| vpc\_type | The type of VPC to attach the project to. Possible options are `base` or `restricted`. | `string` | `""` | no | + +## Outputs + +| Name | Description | +|------|-------------| +| enabled\_apis | VPC Service Control services. | +| kms\_keys | Keys created for the project. | +| project\_id | Project ID. | +| project\_name | Project Name. | +| project\_number | Project number. | +| sa | Project SA email. | + + diff --git a/docs/assets/terraform/4-projects/modules/ml_single_project/main.tf b/docs/assets/terraform/4-projects/modules/ml_single_project/main.tf new file mode 100644 index 00000000..50afa4b1 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_single_project/main.tf @@ -0,0 +1,141 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + enabled_apis = distinct(concat(var.activate_apis, ["billingbudgets.googleapis.com"])) + env_code = element(split("", var.environment), 0) + source_repos = setintersection( + toset(keys(var.app_infra_pipeline_service_accounts)), + toset(keys(var.sa_roles)) + ) + pipeline_roles = var.enable_cloudbuild_deploy ? flatten([ + for repo in local.source_repos : [ + for role in var.sa_roles[repo] : + { + repo = repo + role = role + sa = var.app_infra_pipeline_service_accounts[repo] + } + ] + ]) : [] + + pipeline_kms_sas = var.enable_cloudbuild_deploy ? flatten([ + for repo in keys(var.sa_roles) : [ + var.app_infra_pipeline_service_accounts[repo] + ] + ]) : [] + + network_user_role = var.enable_cloudbuild_deploy ? flatten([ + for repo in local.source_repos : [ + for subnet in var.shared_vpc_subnets : + { + repo = repo + subnet = element(split("/", subnet), index(split("/", subnet), "subnetworks", ) + 1, ) + region = element(split("/", subnet), index(split("/", subnet), "regions") + 1, ) + sa = var.app_infra_pipeline_service_accounts[repo] + } + ] + ]) : [] +} + +module "project" { + source = "terraform-google-modules/project-factory/google" + version = "~> 14.1" + + random_project_id = true + random_project_id_length = 4 + activate_apis = local.enabled_apis + name = var.project_name + org_id = var.org_id + billing_account = var.billing_account + folder_id = var.folder_id + default_service_account = var.default_service_account + + svpc_host_project_id = var.shared_vpc_host_project_id + shared_vpc_subnets = var.shared_vpc_subnets # Optional: To enable subnetting, replace to "module.networking_project.subnetwork_self_link" + + vpc_service_control_attach_enabled = var.vpc_service_control_attach_enabled + vpc_service_control_perimeter_name = var.vpc_service_control_perimeter_name + vpc_service_control_sleep_duration = var.vpc_service_control_sleep_duration + + labels = { + environment = var.environment + application_name = var.application_name + billing_code = var.billing_code + primary_contact = element(split("@", var.primary_contact), 0) + secondary_contact = element(split("@", var.secondary_contact), 0) + business_code = var.business_code + env_code = local.env_code + vpc_type = var.vpc_type + } + budget_alert_pubsub_topic = var.project_budget.alert_pubsub_topic + budget_alert_spent_percents = var.project_budget.alert_spent_percents + budget_amount = var.project_budget.budget_amount + budget_alert_spend_basis = var.project_budget.alert_spend_basis +} + +# Additional roles to the App Infra Pipeline service account +resource "google_project_iam_member" "app_infra_pipeline_sa_roles" { + for_each = { for pr in local.pipeline_roles : "${pr.repo}-${pr.sa}-${pr.role}" => pr } + + project = module.project.project_id + role = each.value.role + member = "serviceAccount:${each.value.sa}" +} + +resource "google_folder_iam_member" "folder_network_viewer" { + for_each = var.app_infra_pipeline_service_accounts + + folder = var.folder_id + role = "roles/compute.networkViewer" + member = "serviceAccount:${each.value}" +} + +resource "google_project_iam_member" "shared_vpc_network_viewer" { + for_each = var.shared_vpc_host_project_id != "" ? toset(local.pipeline_kms_sas) : toset([]) + + project = var.shared_vpc_host_project_id + role = "roles/compute.networkViewer" + member = "serviceAccount:${each.key}" +} + +resource "google_compute_subnetwork_iam_member" "account_role_to_vpc_subnets" { + provider = google-beta + for_each = { for nr in local.network_user_role : "${nr.repo}-${nr.subnet}-${nr.sa}" => nr } + + subnetwork = each.value.subnet + role = "roles/compute.networkUser" + region = each.value.region + project = var.shared_vpc_host_project_id + member = "serviceAccount:${each.value.sa}" +} + +// Add key for project +module "kms_keys" { + source = "../ml_kms_key" + key_rings = var.key_rings + key_rotation_period = var.key_rotation_period + project_name = module.project.project_name + prevent_destroy = var.prevent_destroy +} + +// Add crypto key viewer role to kms environment project +resource "google_project_iam_member" "kms_viewer" { + for_each = var.environment != "common" ? toset(local.pipeline_kms_sas) : toset([]) + project = var.environment_kms_project_id + role = "roles/cloudkms.viewer" + member = "serviceAccount:${each.key}" +} diff --git a/docs/assets/terraform/4-projects/modules/ml_single_project/outputs.tf b/docs/assets/terraform/4-projects/modules/ml_single_project/outputs.tf new file mode 100644 index 00000000..b4aba602 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_single_project/outputs.tf @@ -0,0 +1,45 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "project_id" { + description = "Project ID." + value = module.project.project_id +} + +output "sa" { + description = "Project SA email." + value = module.project.service_account_email +} + +output "project_number" { + description = "Project number." + value = module.project.project_number +} + +output "enabled_apis" { + description = "VPC Service Control services." + value = local.enabled_apis +} + +output "project_name" { + description = "Project Name." + value = module.project.project_name +} + +output "kms_keys" { + description = "Keys created for the project." + value = module.kms_keys.kms_keys +} diff --git a/docs/assets/terraform/4-projects/modules/ml_single_project/variables.tf b/docs/assets/terraform/4-projects/modules/ml_single_project/variables.tf new file mode 100644 index 00000000..d55b3647 --- /dev/null +++ b/docs/assets/terraform/4-projects/modules/ml_single_project/variables.tf @@ -0,0 +1,187 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "org_id" { + description = "The Organization ID." + type = string +} + +variable "folder_id" { + description = "The folder id where project will be created." + type = string +} + +variable "billing_account" { + description = "The ID of the billing account to associated this project with." + type = string +} + +variable "project_suffix" { + description = "The name of the GCP project. Max 16 characters with 3 character business unit code." + type = string +} + +variable "application_name" { + description = "The name of application where GCP resources relate." + type = string +} + +variable "billing_code" { + description = "The code that's used to provide chargeback information." + type = string +} + +variable "primary_contact" { + description = "The primary email contact for the project." + type = string +} + +variable "secondary_contact" { + description = "The secondary email contact for the project." + type = string + default = "" +} + +variable "business_code" { + description = "The code that describes which business unit owns the project." + type = string + default = "abcd" +} + +variable "activate_apis" { + description = "The api to activate for the GCP project." + type = list(string) + default = [] +} + +variable "environment" { + description = "The environment the project belongs to." + type = string +} + +variable "vpc_type" { + description = "The type of VPC to attach the project to. Possible options are `base` or `restricted`." + type = string + default = "" +} + +variable "shared_vpc_host_project_id" { + description = "Shared VPC host project ID." + type = string + default = "" +} + +variable "shared_vpc_subnets" { + description = "List of the shared vpc subnets self links." + type = list(string) + default = [] +} + +variable "vpc_service_control_attach_enabled" { + description = "Whether the project will be attached to a VPC Service Control Perimeter." + type = bool + default = false +} + +variable "vpc_service_control_perimeter_name" { + description = "The name of a VPC Service Control Perimeter to add the created project to." + type = string + default = null +} + +variable "vpc_service_control_sleep_duration" { + description = "The duration to sleep in seconds before adding the project to a shared VPC after the project is added to the VPC Service Control Perimeter." + type = string + default = "5s" +} + +variable "project_budget" { + description = <.iam.gserviceaccount.com`. + - `sa-terraform-env@.iam.gserviceaccount.com` + - `sa-terraform-net@.iam.gserviceaccount.com` + - `sa-terraform-proj@.iam.gserviceaccount.com` + +### Software + +Install the following dependencies: + +- [Google Cloud SDK](https://cloud.google.com/sdk/install) version 469.0.0 or later. +- [Terraform](https://www.terraform.io/downloads.html) version 1.7.5 or later. + +### Google Cloud SDK Configuration + +Terraform must have Application Default Credentials configured, to configure it run: + +```bash +gcloud auth application-default login +``` + +## Directory Layout and Terraform Initialization + +For these instructions we assume that: + +- The foundation was deployed using Cloud Build. +- Every repository, excluding the policies repositories, should be on the `production` branch and `terraform init` should be executed in each one. +- The following layout should exists in your local environment since you will need to make changes in these steps. +If you do not have this layout, please checkout the source repositories for the foundation steps following this layout. + + ```text + gcp-bootstrap + gcp-environments + gcp-networks + gcp-org + gcp-policies + gcp-projects + ``` + +- Also checkout the [terraform-google-enterprise-genai](https://github.com/GoogleCloudPlatform/terraform-google-enterprise-genai) repository at the same level. + +The final layout should look like this: + +```text +gcp-bootstrap +gcp-environments +gcp-networks +gcp-org +gcp-policies +gcp-projects +terraform-google-enterprise-genai +``` + +## Policies + +### Update `gcloud terraform vet` policies + +the first step is to update the `gcloud terraform vet` policies constraints to allow usage of the APIs needed by the Blueprint and add more policies. +The constraints are located in the repository: + +- `gcp-policies` + +All changes below must be made to both repositories: + +**IMPORTANT:** Please note that the steps below are assuming you are checked out on `terraform-google-enterprise-genai/`. + +- Copy `cmek_settings.yaml` from this repository to the policies repository: + +``` bash +cp policy-library/policies/constraints/cmek_settings.yaml ../gcp-policies/policies/constraints/cmek_settings.yaml +``` + +- Copy `network_enable_firewall_logs.yaml` from this repository to the policies repository: + +``` bash +cp policy-library/policies/constraints/network_enable_firewall_logs.yaml ../gcp-policies/policies/constraints/network_enable_firewall_logs.yaml +``` + +- Copy `require_dnssec.yaml` from this repository to the policies repository: + +``` bash +cp policy-library/policies/constraints/require_dnssec.yaml ../gcp-policies/policies/constraints/require_dnssec.yaml +``` + +- On `gcp-policies` change `serviceusage_allow_basic_apis.yaml` and add the following apis: + +```yaml + - "aiplatform.googleapis.com" + - "bigquerymigration.googleapis.com" + - "bigquerystorage.googleapis.com" + - "containerregistry.googleapis.com" + - "dataflow.googleapis.com" + - "dataform.googleapis.com" + - "deploymentmanager.googleapis.com" + - "notebooks.googleapis.com" + - "composer.googleapis.com" + - "containerscanning.googleapis.com" +``` + +Add files to tracked on `gcp-policies` repository, commit and push the code: + +```bash +cd ../gcp-policies + +git add policies/constraints/*.yaml +git commit -m "Add ML policies constraints" +git push origin $(git branch --show-current) +``` + +## 1-org: Create Machine Learning Organization Policies and Organization Level Keys + +This step corresponds to modifications made to `1-org` step on foundation. + +**IMPORTANT:** Please note that the steps below are assuming you are checked out on `terraform-google-enterprise-genai/` and that `gcp-org` repository is checked out on `production` branch. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy Machine Learning modules from this repo to `gcp-org` repository. + +```bash +cp -r 1-org/modules/ml_kms_keyring ../gcp-org/modules +cp -r 1-org/modules/ml-org-policies ../gcp-org/modules +``` + +- Create `ml_ops_org_policy.tf` file on `gcp-org/envs/shared` path: + +```bash +cp docs/assets/terraform/1-org/ml_ops_org_policy.tf ../gcp-org/envs/shared +``` + +- Create `ml_key_rings.tf` file on `gcp-org/envs/shared` path: + +```bash +cp docs/assets/terraform/1-org/ml_key_rings.tf ../gcp-org/envs/shared +``` + +- Edit `gcp-org/envs/shared/remote.tf` and add the following value to `locals`: + +```terraform +projects_step_terraform_service_account_email = data.terraform_remote_state.bootstrap.outputs.projects_step_terraform_service_account_email +``` + +- Edit `gcp-org/envs/shared/variables.tf` and add the following variables: + +```terraform +variable "keyring_regions" { + description = "Regions to create keyrings in" + type = list(string) + default = [ + "us-central1", + "us-east4" + ] +} + +variable "keyring_name" { + description = "Name to be used for KMS Keyring" + type = string + default = "ml-org-keyring" +} +``` + +- Edit `gcp-org/envs/shared/outputs.tf` and add the following output: + +```terraform +output "key_rings" { + description = "Keyring Names created" + value = module.kms_keyring.key_rings +} +``` + +Add files to git on `gcp-org`, commit and push code: + +```bash +cd ../gcp-org + +git add . + +git commit -m "Add ML org policies and Org-level key" +git push origin production +``` + +## 2-environment: Create environment level logging keys, logging project and logging bucket + +This step corresponds to modifications made to `2-environment` step on foundation. + +Please note that the steps below are assuming you are checked out on `terraform-google-enterprise-genai/`. + +```bash +cd ../terraform-google-enterprise-genai +``` + +### `development` branch + +- Go to `gcp-environments` repository, and check out on `development` branch. + +```bash +cd ../gcp-environments + +git checkout development +``` + +- Return to `terraform-google-enterprise-genai` repo. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy Machine Learning modules from this repo to `gcp-environments` repository. + +```bash +cp -r 2-environments/modules/ml_kms_keyring ../gcp-environments/modules +``` + +- Create `ml_key_rings.tf` file on `gcp-environments/modules/env_baseline` path: + +```bash +cp docs/assets/terraform/2-environments/ml_key_rings.tf ../gcp-environments/modules/env_baseline +``` + +- Create `ml_logging.tf` file on `gcp-environments/modules/env_baseline` path: + +```bash +cp docs/assets/terraform/2-environments/ml_logging.tf ../gcp-environments/modules/env_baseline +``` + +- On `gcp-environments/modules/env_baseline/variables.tf` add the following variables: + +```terraform +variable "keyring_name" { + description = "Name to be used for KMS Keyring" + type = string + default = "ml-env-keyring" +} + +variable "keyring_regions" { + description = "Regions to create keyrings in" + type = list(string) + default = [ + "us-central1", + "us-east4" + ] +} + +variable "kms_prevent_destroy" { + description = "Wheter to prevent keyring and keys destruction. Must be set to false if the user wants to disable accidental terraform deletions protection." + type = bool + default = true +} + +variable "gcs_bucket_prefix" { + description = "Bucket Prefix" + type = string + default = "bkt" +} + +variable "gcs_logging_bucket_location" { + description = "Location of environment logging bucket" + type = string + default = "us-central1" +} + +variable "gcs_logging_retention_period" { + description = "Retention configuration for environment logging bucket" + type = object({ + is_locked = bool + retention_period_days = number + }) + default = null +} + +variable "gcs_logging_key_rotation_period" { + description = "Rotation period in seconds to be used for KMS Key" + type = string + default = "7776000s" +} +``` + +- On `gcp-environments/modules/env_baseline/variables.tf` add the following field to `project_budget` specification: + +```terraform +logging_budget_amount = optional(number, 1000) +logging_alert_spent_percents = optional(list(number), [1.2]) +logging_alert_pubsub_topic = optional(string, null) +logging_budget_alert_spend_basis = optional(string, "FORECASTED_SPEND") +``` + +This will result in a variable similar to the variable specified below: + +```terraform +variable "project_budget" { + description = < Note: you might receive an error telling you that this is against an organization policy, this can happen because of the propagation time from the change made to the organization policy (propagation time is tipically 2 minutes, but can take 7 minutes or longer). If this happens, wait some minutes and try again + +6. Delete the change made on the first step to the organization policy, this will make the project inherit parent policies. + + ```bash + gcloud org-policies delete iam.allowedPolicyMemberDomains --project=$ENV_LOG_PROJECT_ID + ``` + +##### `non-production` environment configuration + +1. Configure the following variable below with the value of `gcp-environments` repository path. + + ```bash + export GCP_ENVIRONMENTS_PATH=INSERT_YOUR_PATH_HERE + ``` + + Make sure your git is checked out to the `non-production` branch by running `git checkout nonproduction` on `GCP_ENVIRONMENTS_PATH`. + + ```bash + (cd $GCP_ENVIRONMENTS_PATH && git checkout nonproduction) + ``` + +2. Retrieve the bucket name and project id from terraform outputs. + + ```bash + export ENV_LOG_BUCKET_NAME=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/nonproduction" output -raw env_log_bucket_name) + export ENV_LOG_PROJECT_ID=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/nonproduction" output -raw env_log_project_id) + ``` + +3. Validate the variable values. + + ```bash + echo env_log_project_id=$ENV_LOG_PROJECT_ID + echo env_log_bucket_name=$ENV_LOG_BUCKET_NAME + ``` + +4. Reset your org policy for the logging project by running the following command. + + ```bash + gcloud org-policies reset iam.allowedPolicyMemberDomains --project=$ENV_LOG_PROJECT_ID + ``` + +5. Assign `roles/storage.objectCreator` role to `cloud-storage-analytics@google.com` group. + + ```bash + gcloud storage buckets add-iam-policy-binding gs://$ENV_LOG_BUCKET_NAME --member="group:cloud-storage-analytics@google.com" --role="roles/storage.objectCreator" + ``` + + > Note: you might receive an error telling you that this is against an organization policy, this can happen because of the propagation time from the change made to the organization policy (propagation time is tipically 2 minutes, but can take 7 minutes or longer). If this happens, wait some minutes and try again + +6. Delete the change made on the first step to the organization policy, this will make the project inherit parent policies. + + ```bash + gcloud org-policies delete iam.allowedPolicyMemberDomains --project=$ENV_LOG_PROJECT_ID + ``` + +##### `production` environment configuration + +1. Configure the following variable below with the value of `gcp-environments` repository path. + + ```bash + export GCP_ENVIRONMENTS_PATH=INSERT_YOUR_PATH_HERE + ``` + + Make sure your git is checked out to the `production` branch by running `git checkout production` on `GCP_ENVIRONMENTS_PATH`. + + ```bash + (cd $GCP_ENVIRONMENTS_PATH && git checkout production) + ``` + +2. Retrieve the bucket name and project id from terraform outputs. + + ```bash + export ENV_LOG_BUCKET_NAME=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/production" output -raw env_log_bucket_name) + export ENV_LOG_PROJECT_ID=$(terraform -chdir="$GCP_ENVIRONMENTS_PATH/envs/production" output -raw env_log_project_id) + ``` + +3. Validate the variable values. + + ```bash + echo env_log_project_id=$ENV_LOG_PROJECT_ID + echo env_log_bucket_name=$ENV_LOG_BUCKET_NAME + ``` + +4. Reset your org policy for the logging project by running the following command. + + ```bash + gcloud org-policies reset iam.allowedPolicyMemberDomains --project=$ENV_LOG_PROJECT_ID + ``` + +5. Assign `roles/storage.objectCreator` role to `cloud-storage-analytics@google.com` group. + + ```bash + gcloud storage buckets add-iam-policy-binding gs://$ENV_LOG_BUCKET_NAME --member="group:cloud-storage-analytics@google.com" --role="roles/storage.objectCreator" + ``` + + > Note: you might receive an error telling you that this is against an organization policy, this can happen because of the propagation time from the change made to the organization policy (propagation time is tipically 2 minutes, but can take 7 minutes or longer). If this happens, wait some minutes and try again + +6. Delete the change made on the first step to the organization policy, this will make the project inherit parent policies. + + ```bash + gcloud org-policies delete iam.allowedPolicyMemberDomains --project=$ENV_LOG_PROJECT_ID + ``` + +#### Option 2: Use Google Cloud Console to disable/enable organization policy constraint + +Proceed with these steps only if `Option 1` is not chosen. + +1. On `ml_logging.tf` locate the following lines and uncomment them: + + ```terraform + resource "google_storage_bucket_iam_member" "bucket_logging" { + bucket = google_storage_bucket.log_bucket.name + role = "roles/storage.objectCreator" + member = "group:cloud-storage-analytics@google.com" + } + ``` + +2. Under `IAM & Admin`, select `Organization Policies`. Search for "Domain Restricted Sharing". + + ![list-policy](../2-environments/imgs/list-policy.png) + +3. Select 'Manage Policy'. This directs you to the Domain Restricted Sharing Edit Policy page. It will be set at 'Inherit parent's policy'. Change this to 'Google-managed default'. + + ![edit-policy](../2-environments/imgs/edit-policy.png) + +4. Follow the instructions on checking out `development`, `non-production` & `production` branches. Once environments terraform code has successfully applied, edit the policy again and select 'Inherit parent's policy' and Click `SET POLICY`. + +After making these modifications, you can follow the README.md procedure for `2-environment` step on foundation, make sure you **change the organization policy after running the steps on foundation**. + +## 3-network: Configure private DNS zone for Vertex Workbench Instances, Enable NAT and Attach projects to perimeter + +This step corresponds to modifications made to `3-networks-dual-svpc` step on foundation. + +Please note that the steps below are assuming you are checked out on `terraform-google-enterprise-genai/`. + +```bash +cd ../terraform-google-enterprise-genai +``` + +### `development` branch on `gcp-networks` + +- Go to `gcp-networks` repository, and check out on `development` branch. + +```bash +cd ../gcp-networks + +git checkout development +``` + +#### Private DNS zone configuration (dev) + +- Return to `terraform-google-enterprise-genai` repo. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy DNS notebook network module from this repo to `gcp-networks` repository. + +```bash +cp -r 3-networks-dual-svpc/modules/ml_dns_notebooks ../gcp-networks/modules +``` + +- Create a file named `ml_dns_notebooks.tf` on path `gcp-networks/modules/base_env`: + +```bash +cp docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf ../gcp-networks/modules/base_env +``` + +Commit and push files to git repo. + +```bash +cd ../gcp-networks + +git add . + +git commit -m "Create DNS notebook configuration" + +git push origin development +``` + +#### Enabling NAT, Attaching projects to Service Perimeter and Creating custom firewall rules (dev) + +Create `gcp-networks/modules/base_env/data.tf` file with the following content: + +```terraform +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +data "google_netblock_ip_ranges" "legacy_health_checkers" { + range_type = "legacy-health-checkers" +} + +data "google_netblock_ip_ranges" "health_checkers" { + range_type = "health-checkers" +} + +// Cloud IAP's TCP forwarding netblock +data "google_netblock_ip_ranges" "iap_forwarders" { + range_type = "iap-forwarders" +} +``` + +On `gcp-networks/modules/restricted_shared_vpc/variables.tf` add the following variables: + +```terraform +variable "perimeter_projects" { + description = "A list of project numbers to be added to the service perimeter" + type = list(number) + default = [] +} + +variable "allow_all_egress_ranges" { + description = "List of network ranges to which all egress traffic will be allowed" + default = null +} + +variable "allow_all_ingress_ranges" { + description = "List of network ranges from which all ingress traffic will be allowed" + default = null +} +``` + +On `gcp-networks/modules/base_env/remote.tf`: + +1. Add the env remote state, by adding the following terraform code to the file: + + ```terraform + data "terraform_remote_state" "env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } + } + ``` + +2. Edit `locals` and add the following fields: + + ```terraform + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + ``` + +3. The final result will contain existing locals and the added ones, it should look similar to the code below: + + ```terraform + locals { + restricted_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_id + restricted_project_number = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_number + base_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].base_shared_vpc_project_id + interconnect_project_number = data.terraform_remote_state.org.outputs.interconnect_project_number + dns_hub_project_id = data.terraform_remote_state.org.outputs.dns_hub_project_id + organization_service_account = data.terraform_remote_state.bootstrap.outputs.organization_step_terraform_service_account_email + networks_service_account = data.terraform_remote_state.bootstrap.outputs.networks_step_terraform_service_account_email + projects_service_account = data.terraform_remote_state.bootstrap.outputs.projects_step_terraform_service_account_email + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + } + ``` + +##### Adding projects to service perimeter (dev) + +On `gcp-networks/modules/restricted_shared_vpc/service_control.tf`, modify the terraform module called **regular_service_perimeter** and add the following module field to `resources`: + +```terraform +distinct(concat([var.project_number], var.perimeter_projects)) +``` + +This shall result in a module similar to the code below: + +```terraform +module "regular_service_perimeter" { + source = "terraform-google-modules/vpc-service-controls/google//modules/regular_service_perimeter" + version = "~> 4.0" + + policy = var.access_context_manager_policy_id + perimeter_name = local.perimeter_name + description = "Default VPC Service Controls perimeter" + resources = distinct(concat([var.project_number], var.perimeter_projects)) + access_levels = [module.access_level_members.name] + + restricted_services = var.restricted_services + vpc_accessible_services = ["RESTRICTED-SERVICES"] + + ingress_policies = var.ingress_policies + egress_policies = var.egress_policies + + depends_on = [ + time_sleep.wait_vpc_sc_propagation + ] +} +``` + +##### Creating "allow all ingress ranges" and "allow all egress ranges" firewall rules (dev) + +On `gcp-networks/modules/restricted_shared_vpc/firewall.tf` add the following firewall rules by adding the terraform code below to the file: + +```terraform +resource "google_compute_firewall" "allow_all_egress" { + count = var.allow_all_egress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-e-a-all-all-all" + network = module.main.network_name + project = var.project_id + direction = "EGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + destination_ranges = var.allow_all_egress_ranges +} + +resource "google_compute_firewall" "allow_all_ingress" { + count = var.allow_all_ingress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-i-a-all" + network = module.main.network_name + project = var.project_id + direction = "INGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + source_ranges = var.allow_all_ingress_ranges +} +``` + +##### Changes to restricted shared VPC (dev) + +On `gcp-networks/modules/base_env/main.tf` edit the terraform module named **restricted_shared_vpc** and add the following fields to it: + +```terraform +allow_all_ingress_ranges = concat(data.google_netblock_ip_ranges.health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.legacy_health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.iap_forwarders.cidr_blocks_ipv4) +allow_all_egress_ranges = ["0.0.0.0/0"] + +nat_enabled = true +nat_num_addresses_region1 = 1 +nat_num_addresses_region2 = 1 + +perimeter_projects = [local.logging_env_project_number, local.kms_env_project_number] +``` + +Commit all changes and push to the current branch. + +```bash +git add . +git commit -m "Create custom fw rules, enable nat, configure dns and service perimeter" + +git push origin development +``` + +### `nonproduction` branch on `gcp-networks` + +- Go to `gcp-networks` repository, and check out on `nonproduction` branch. + +```bash +cd ../gcp-networks + +git checkout nonproduction +``` + +#### Private DNS zone configuration (non-production) + +- Return to `terraform-google-enterprise-genai` repo. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy DNS notebook network module from this repo to `gcp-networks` repository. + +```bash +cp -r 3-networks-dual-svpc/modules/ml_dns_notebooks ../gcp-networks/modules +``` + +- Create a file named `ml_dns_notebooks.tf` on path `gcp-networks/modules/base_env`: + +```bash +cp docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf ../gcp-networks/modules/base_env +``` + +Commit and push files to git repo. + +```bash +cd ../gcp-networks + +git add . + +git commit -m "Create DNS notebook configuration" + +git push origin nonproduction +``` + +#### Enabling NAT, Attaching projects to Service Perimeter and Creating custom firewall rules (non-production) + +Create `gcp-networks/modules/base_env/data.tf` file with the following content: + +```terraform +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +data "google_netblock_ip_ranges" "legacy_health_checkers" { + range_type = "legacy-health-checkers" +} + +data "google_netblock_ip_ranges" "health_checkers" { + range_type = "health-checkers" +} + +// Cloud IAP's TCP forwarding netblock +data "google_netblock_ip_ranges" "iap_forwarders" { + range_type = "iap-forwarders" +} +``` + +On `gcp-networks/modules/restricted_shared_vpc/variables.tf` add the following variables: + +```terraform +variable "perimeter_projects" { + description = "A list of project numbers to be added to the service perimeter" + type = list(number) + default = [] +} + +variable "allow_all_egress_ranges" { + description = "List of network ranges to which all egress traffic will be allowed" + default = null +} + +variable "allow_all_ingress_ranges" { + description = "List of network ranges from which all ingress traffic will be allowed" + default = null +} +``` + +On `gcp-networks/modules/base_env/remote.tf`: + +1. Add the env remote state, by adding the following terraform code to the file: + + ```terraform + data "terraform_remote_state" "env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } + } + ``` + +2. Edit `locals` and add the following fields: + + ```terraform + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + ``` + +3. The final result will contain existing locals and the added ones, it should look similar to the code below: + + ```terraform + locals { + restricted_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_id + restricted_project_number = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_number + base_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].base_shared_vpc_project_id + interconnect_project_number = data.terraform_remote_state.org.outputs.interconnect_project_number + dns_hub_project_id = data.terraform_remote_state.org.outputs.dns_hub_project_id + organization_service_account = data.terraform_remote_state.bootstrap.outputs.organization_step_terraform_service_account_email + networks_service_account = data.terraform_remote_state.bootstrap.outputs.networks_step_terraform_service_account_email + projects_service_account = data.terraform_remote_state.bootstrap.outputs.projects_step_terraform_service_account_email + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + } + ``` + +##### Adding projects to service perimeter (non-production) + +On `gcp-networks/modules/restricted_shared_vpc/service_control.tf`, modify the terraform module called **regular_service_perimeter** and add the following module field to `resources`: + +```terraform +distinct(concat([var.project_number], var.perimeter_projects)) +``` + +This shall result in a module similar to the code below: + +```terraform +module "regular_service_perimeter" { + source = "terraform-google-modules/vpc-service-controls/google//modules/regular_service_perimeter" + version = "~> 4.0" + + policy = var.access_context_manager_policy_id + perimeter_name = local.perimeter_name + description = "Default VPC Service Controls perimeter" + resources = distinct(concat([var.project_number], var.perimeter_projects)) + access_levels = [module.access_level_members.name] + + restricted_services = var.restricted_services + vpc_accessible_services = ["RESTRICTED-SERVICES"] + + ingress_policies = var.ingress_policies + egress_policies = var.egress_policies + + depends_on = [ + time_sleep.wait_vpc_sc_propagation + ] +} +``` + +##### Creating "allow all ingress ranges" and "allow all egress ranges" firewall rules (non-production) + +On `gcp-networks/modules/restricted_shared_vpc/firewall.tf` add the following firewall rules by adding the terraform code below to the file: + +```terraform +resource "google_compute_firewall" "allow_all_egress" { + count = var.allow_all_egress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-e-a-all-all-all" + network = module.main.network_name + project = var.project_id + direction = "EGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + destination_ranges = var.allow_all_egress_ranges +} + +resource "google_compute_firewall" "allow_all_ingress" { + count = var.allow_all_ingress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-i-a-all" + network = module.main.network_name + project = var.project_id + direction = "INGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + source_ranges = var.allow_all_ingress_ranges +} +``` + +##### Changes to restricted shared VPC (non-production) + +On `gcp-networks/modules/base_env/main.tf` edit the terraform module named **restricted_shared_vpc** and add the following fields to it: + +```terraform +allow_all_ingress_ranges = concat(data.google_netblock_ip_ranges.health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.legacy_health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.iap_forwarders.cidr_blocks_ipv4) +allow_all_egress_ranges = ["0.0.0.0/0"] + +nat_enabled = true +nat_num_addresses_region1 = 1 +nat_num_addresses_region2 = 1 + +perimeter_projects = [local.logging_env_project_number, local.kms_env_project_number] +``` + +Commit all changes and push to the current branch. + +```bash +git add . +git commit -m "Create custom fw rules, enable nat, configure dns and service perimeter" + +git push origin nonproduction +``` + +### `production` branch on `gcp-networks` + +- Go to `gcp-networks` repository, and check out on `production` branch. + +```bash +cd ../gcp-networks + +git checkout production +``` + +#### Private DNS zone configuration (production) + +- Return to `terraform-google-enterprise-genai` repo. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy DNS notebook network module from this repo to `gcp-networks` repository. + +```bash +cp -r 3-networks-dual-svpc/modules/ml_dns_notebooks ../gcp-networks/modules +``` + +- Create a file named `ml_dns_notebooks.tf` on path `gcp-networks/modules/base_env`: + +```bash +cp docs/assets/terraform/3-networks-dual-svpc/ml_dns_notebooks.tf ../gcp-networks/modules/base_env +``` + +Commit and push files to git repo. + +```bash +cd ../gcp-networks + +git add . + +git commit -m "Create DNS notebook configuration" + +git push origin production +``` + +#### Enabling NAT, Attaching projects to Service Perimeter and Creating custom firewall rules (production) + +Create `gcp-networks/modules/base_env/data.tf` file with the following content: + +```terraform +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +data "google_netblock_ip_ranges" "legacy_health_checkers" { + range_type = "legacy-health-checkers" +} + +data "google_netblock_ip_ranges" "health_checkers" { + range_type = "health-checkers" +} + +// Cloud IAP's TCP forwarding netblock +data "google_netblock_ip_ranges" "iap_forwarders" { + range_type = "iap-forwarders" +} +``` + +On `gcp-networks/modules/restricted_shared_vpc/variables.tf` add the following variables: + +```terraform +variable "perimeter_projects" { + description = "A list of project numbers to be added to the service perimeter" + type = list(number) + default = [] +} + +variable "allow_all_egress_ranges" { + description = "List of network ranges to which all egress traffic will be allowed" + default = null +} + +variable "allow_all_ingress_ranges" { + description = "List of network ranges from which all ingress traffic will be allowed" + default = null +} +``` + +On `gcp-networks/modules/base_env/remote.tf`: + +1. Add the env remote state, by adding the following terraform code to the file: + + ```terraform + data "terraform_remote_state" "env" { + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/environments/${var.env}" + } + } + ``` + +2. Edit `locals` and add the following fields: + + ```terraform + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + ``` + +3. The final result will contain existing locals and the added ones, it should look similar to the code below: + + ```terraform + locals { + restricted_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_id + restricted_project_number = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].restricted_shared_vpc_project_number + base_project_id = data.terraform_remote_state.org.outputs.shared_vpc_projects[var.env].base_shared_vpc_project_id + interconnect_project_number = data.terraform_remote_state.org.outputs.interconnect_project_number + dns_hub_project_id = data.terraform_remote_state.org.outputs.dns_hub_project_id + organization_service_account = data.terraform_remote_state.bootstrap.outputs.organization_step_terraform_service_account_email + networks_service_account = data.terraform_remote_state.bootstrap.outputs.networks_step_terraform_service_account_email + projects_service_account = data.terraform_remote_state.bootstrap.outputs.projects_step_terraform_service_account_email + logging_env_project_number = data.terraform_remote_state.env.outputs.env_log_project_number + kms_env_project_number = data.terraform_remote_state.env.outputs.env_kms_project_number + } + ``` + +##### Adding projects to service perimeter (production) + +On `gcp-networks/modules/restricted_shared_vpc/service_control.tf`, modify the terraform module called **regular_service_perimeter** and add the following module field to `resources`: + +```terraform +distinct(concat([var.project_number], var.perimeter_projects)) +``` + +This shall result in a module similar to the code below: + +```terraform +module "regular_service_perimeter" { + source = "terraform-google-modules/vpc-service-controls/google//modules/regular_service_perimeter" + version = "~> 4.0" + + policy = var.access_context_manager_policy_id + perimeter_name = local.perimeter_name + description = "Default VPC Service Controls perimeter" + resources = distinct(concat([var.project_number], var.perimeter_projects)) + access_levels = [module.access_level_members.name] + + restricted_services = var.restricted_services + vpc_accessible_services = ["RESTRICTED-SERVICES"] + + ingress_policies = var.ingress_policies + egress_policies = var.egress_policies + + depends_on = [ + time_sleep.wait_vpc_sc_propagation + ] +} +``` + +##### Creating "allow all ingress ranges" and "allow all egress ranges" firewall rules (production) + +On `gcp-networks/modules/restricted_shared_vpc/firewall.tf` add the following firewall rules by adding the terraform code below to the file: + +```terraform +resource "google_compute_firewall" "allow_all_egress" { + count = var.allow_all_egress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-e-a-all-all-all" + network = module.main.network_name + project = var.project_id + direction = "EGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + destination_ranges = var.allow_all_egress_ranges +} + +resource "google_compute_firewall" "allow_all_ingress" { + count = var.allow_all_ingress_ranges != null ? 1 : 0 + + name = "fw-${var.environment_code}-shared-base-1000-i-a-all" + network = module.main.network_name + project = var.project_id + direction = "INGRESS" + priority = 1000 + + dynamic "log_config" { + for_each = var.firewall_enable_logging == true ? [{ + metadata = "INCLUDE_ALL_METADATA" + }] : [] + + content { + metadata = log_config.value.metadata + } + } + + allow { + protocol = "all" + } + + source_ranges = var.allow_all_ingress_ranges +} +``` + +##### Changes to restricted shared VPC (production) + +On `gcp-networks/modules/base_env/main.tf` edit the terraform module named **restricted_shared_vpc** and add the following fields to it: + +```terraform +allow_all_ingress_ranges = concat(data.google_netblock_ip_ranges.health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.legacy_health_checkers.cidr_blocks_ipv4, data.google_netblock_ip_ranges.iap_forwarders.cidr_blocks_ipv4) +allow_all_egress_ranges = ["0.0.0.0/0"] + +nat_enabled = true +nat_num_addresses_region1 = 1 +nat_num_addresses_region2 = 1 + +perimeter_projects = [local.logging_env_project_number, local.kms_env_project_number] +``` + +Commit all changes and push to the current branch. + +```bash +git add . +git commit -m "Create custom fw rules, enable nat, configure dns and service perimeter" + +git push origin production +``` + +## 4-projects: Create Service Catalog and Artifacts Shared projects and Machine Learning Projects + +This step corresponds to modifications made to `4-projects` step on foundation. + +Please note that the steps below are assuming you are checked out on `terraform-google-enterprise-genai/`. + +```bash +cd ../terraform-google-enterprise-genai +``` + +In this tutorial, we are using `ml_business_unit` as an example. + +You need to manually plan and apply only once the `ml_business_unit/shared`. + +### Manually applying `shared` + +- Go to `gcp-projects` repository and checkout to `plan` branch. + +```bash +cd ../gcp-projects + +git checkout plan +``` + +- Return to GenAI repository. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy `ml_business_unit` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/ml_business_unit ../gcp-projects +``` + +- Add modules to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/modules/* ../gcp-projects/modules +``` + +- Add `tfvars` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/*.example.tfvars ../gcp-projects +``` + +- Go to `gcp-projects` repository. + +```bash +cd ../gcp-projects +``` + +- Update project backend by retrieving it's value from `0-bootstrap` and applying it to `backend.tf`. + +```bash +export PROJECT_BACKEND=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw projects_gcs_bucket_tfstate) + +for file in $(find . -name backend.tf); do sed -i "s/UPDATE_PROJECTS_BACKEND/$PROJECT_BACKEND/" $file; done +``` + +- Retrieve projects step service account e-mail. + +```bash +export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw projects_step_terraform_service_account_email) +echo ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT} +``` + +- Retrieve cloud build project id. + +```bash +export CLOUD_BUILD_PROJECT_ID=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw cloudbuild_project_id) +echo ${CLOUD_BUILD_PROJECT_ID} +``` + +- Rename `auto.example.tfvars` to `auto.tfvars`. + +```bash +mv common.auto.example.tfvars common.auto.tfvars +mv shared.auto.example.tfvars shared.auto.tfvars +mv development.auto.example.tfvars development.auto.tfvars +mv non-production.auto.example.tfvars non-production.auto.tfvars +mv production.auto.example.tfvars production.auto.tfvars +``` + +- Update REMOTE_STATE_BUCKET value. + +```bash +export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate) +echo "remote_state_bucket = ${remote_state_bucket}" + +sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars +``` + +- Commit the changes. + +```bash +git add . + +git commit -m "Create ML Business Unit" +``` + +- Log into gcloud using service account impersonation and then set your configuration: + +```bash +gcloud auth application-default login --impersonate-service-account=${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT} +``` + +- Run `init` and `plan` and review output for environment shared. + +```bash +./tf-wrapper.sh init shared +./tf-wrapper.sh plan shared +``` + +- Run `validate` and check for violations. + +```bash +./tf-wrapper.sh validate shared $(pwd)/../gcp-policies ${CLOUD_BUILD_PROJECT_ID} +``` + +- Run `apply` shared. + +```bash +./tf-wrapper.sh apply shared +``` + +This will create the artifacts and service catalog projects under `common` folder and configure the Machine Learning business unit infra pipeline. + +Push plan branch to remote. + +```bash +git push origin plan +``` + +### `development` branch on `gcp-projects` + +This will create the machine learning development environment. A Machine Learning project will be hosted under a folder. + +- Go to `gcp-projects` repository and checkout to `plan` branch. + +```bash +cd ../gcp-projects + +git checkout development +``` + +- Return to GenAI repository. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy `ml_business_unit` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/ml_business_unit ../gcp-projects +``` + +- Add modules to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/modules/* ../gcp-projects/modules +``` + +- Add `tfvars` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/*.example.tfvars ../gcp-projects +``` + +- Go to `gcp-projects` repository. + +```bash +cd ../gcp-projects +``` + +- Rename `auto.example.tfvars` to `auto.tfvars`. + +```bash +mv common.auto.example.tfvars common.auto.tfvars +mv shared.auto.example.tfvars shared.auto.tfvars +mv development.auto.example.tfvars development.auto.tfvars +mv non-production.auto.example.tfvars non-production.auto.tfvars +mv production.auto.example.tfvars production.auto.tfvars +``` + +- Update REMOTE_STATE_BUCKET value. + +```bash +export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate) +echo "remote_state_bucket = ${remote_state_bucket}" + +sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars +``` + +- Update project backend by retrieving it's value from `0-bootstrap` and applying it to `backend.tf`. + +```bash +export PROJECT_BACKEND=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw projects_gcs_bucket_tfstate) + +for file in $(find . -name backend.tf); do sed -i "s/UPDATE_PROJECTS_BACKEND/$PROJECT_BACKEND/" $file; done +``` + +- Commit and push. + +```bash +git add . +git commit -m "Initialize ML environment" + +git push origin development +``` + +### `nonproduction` branch on `gcp-projects` + +This will create the machine learning nonproduction environment. A Machine Learning project will be hosted under a folder. + +- Go to `gcp-projects` repository and checkout to `plan` branch. + +```bash +cd ../gcp-projects + +git checkout nonproduction +``` + +- Return to GenAI repository. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy `ml_business_unit` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/ml_business_unit ../gcp-projects +``` + +- Add modules to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/modules/* ../gcp-projects/modules +``` + +- Add `tfvars` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/*.example.tfvars ../gcp-projects +``` + +- Go to `gcp-projects` repository. + +```bash +cd ../gcp-projects +``` + +- Rename `auto.example.tfvars` to `auto.tfvars`. + +```bash +mv common.auto.example.tfvars common.auto.tfvars +mv shared.auto.example.tfvars shared.auto.tfvars +mv development.auto.example.tfvars development.auto.tfvars +mv non-production.auto.example.tfvars non-production.auto.tfvars +mv production.auto.example.tfvars production.auto.tfvars +``` + +- Update REMOTE_STATE_BUCKET value. + +```bash +export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate) +echo "remote_state_bucket = ${remote_state_bucket}" + +sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars +``` + +- Update project backend by retrieving it's value from `0-bootstrap` and applying it to `backend.tf`. + +```bash +export PROJECT_BACKEND=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw projects_gcs_bucket_tfstate) + +for file in $(find . -name backend.tf); do sed -i "s/UPDATE_PROJECTS_BACKEND/$PROJECT_BACKEND/" $file; done +``` + +- Commit and push. + +```bash +git add . +git commit -m "Initialize ML environment" + +git push origin nonproduction +``` + +### `production` branch on `gcp-projects` + +This will create the machine learning production environment. A Machine Learning project will be hosted under a folder. + +- Go to `gcp-projects` repository and checkout to `plan` branch. + +```bash +cd ../gcp-projects + +git checkout production +``` + +- Return to GenAI repository. + +```bash +cd ../terraform-google-enterprise-genai +``` + +- Copy `ml_business_unit` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/ml_business_unit ../gcp-projects +``` + +- Remove shared directory on `ml_business_unit` on the `gcp-projects` repository. + +```bash +rm -rf ../gcp-projects/ml_business_unit/shared +``` + +- Add modules to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/modules/* ../gcp-projects/modules +``` + +- Add `tfvars` to the `gcp-projects` repository. + +```bash +cp -r docs/assets/terraform/4-projects/*.example.tfvars ../gcp-projects +``` + +- Go to `gcp-projects` repository. + +```bash +cd ../gcp-projects +``` + +- Rename `auto.example.tfvars` to `auto.tfvars`. + +```bash +mv common.auto.example.tfvars common.auto.tfvars +mv shared.auto.example.tfvars shared.auto.tfvars +mv development.auto.example.tfvars development.auto.tfvars +mv non-production.auto.example.tfvars non-production.auto.tfvars +mv production.auto.example.tfvars production.auto.tfvars +``` + +- Update REMOTE_STATE_BUCKET value. + +```bash +export remote_state_bucket=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw gcs_bucket_tfstate) +echo "remote_state_bucket = ${remote_state_bucket}" + +sed -i'' -e "s/REMOTE_STATE_BUCKET/${remote_state_bucket}/" ./common.auto.tfvars +``` + +- Update project backend by retrieving it's value from `0-bootstrap` and applying it to `backend.tf`. + +```bash +export PROJECT_BACKEND=$(terraform -chdir="../gcp-bootstrap/envs/shared" output -raw projects_gcs_bucket_tfstate) + +for file in $(find . -name backend.tf); do sed -i "s/UPDATE_PROJECTS_BACKEND/$PROJECT_BACKEND/" $file; done +``` + +- Commit and push. + +```bash +git add . +git commit -m "Initialize ML environment" + +git push origin production +``` From cbf035be6ffc646d440a932e7bce66f653786357 Mon Sep 17 00:00:00 2001 From: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> Date: Tue, 11 Jun 2024 16:08:11 -0300 Subject: [PATCH 3/4] chore: rename 'ml_infra_project' to 'ml_infra_projects' for consistency (#44) * chore: adjust naming * adjust outputs --- .../business_unit_3/shared/ml_infra_projects.tf | 2 +- 4-projects/business_unit_3/shared/outputs.tf | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/4-projects/business_unit_3/shared/ml_infra_projects.tf b/4-projects/business_unit_3/shared/ml_infra_projects.tf index acc9abea..431b0c2a 100644 --- a/4-projects/business_unit_3/shared/ml_infra_projects.tf +++ b/4-projects/business_unit_3/shared/ml_infra_projects.tf @@ -14,7 +14,7 @@ * limitations under the License. */ -module "ml_infra_project" { +module "ml_infra_projects" { source = "../../modules/ml_infra_projects" org_id = local.org_id diff --git a/4-projects/business_unit_3/shared/outputs.tf b/4-projects/business_unit_3/shared/outputs.tf index 2d0e696d..88c5d34c 100644 --- a/4-projects/business_unit_3/shared/outputs.tf +++ b/4-projects/business_unit_3/shared/outputs.tf @@ -65,30 +65,30 @@ output "enable_cloudbuild_deploy" { output "service_catalog_project_id" { description = "Service Catalog Project ID." - value = module.ml_infra_project.service_catalog_project_id + value = module.ml_infra_projects.service_catalog_project_id } output "common_artifacts_project_id" { description = "App Infra Artifacts Project ID" - value = module.ml_infra_project.common_artifacts_project_id + value = module.ml_infra_projects.common_artifacts_project_id } output "service_catalog_repo_name" { description = "The name of the Service Catalog repository" - value = module.ml_infra_project.service_catalog_repo_name + value = module.ml_infra_projects.service_catalog_repo_name } output "service_catalog_repo_id" { description = "ID of the Service Catalog repository" - value = module.ml_infra_project.service_catalog_repo_id + value = module.ml_infra_projects.service_catalog_repo_id } output "artifacts_repo_name" { description = "The name of the Artifacts repository" - value = module.ml_infra_project.artifacts_repo_name + value = module.ml_infra_projects.artifacts_repo_name } output "artifacts_repo_id" { description = "ID of the Artifacts repository" - value = module.ml_infra_project.artifacts_repo_id + value = module.ml_infra_projects.artifacts_repo_id } From 7472009670f483205c4dce85398ae313cd3d48cd Mon Sep 17 00:00:00 2001 From: Caetano Colin <164910343+caetano-colin@users.noreply.github.com> Date: Tue, 11 Jun 2024 16:09:42 -0300 Subject: [PATCH 4/4] feat(example): Add Generative AI example using RAG and Vector Search (#46) * feat(module): add genai example * feat: add README.md * edit README.md * add known issues * add fix * add service agent role assignment * update terraform.tfvars * Update README with VPC-SC Instruction * fix render * Update module * add regional bucket and fix deployed index access * add Google Header to file * update README.md with terraform docs * lint fixes * implement PR review changes * update host_vpc projectid --- examples/genai-rag-multimodal/README.md | 139 +++ .../multimodal_rag_langchain.ipynb | 1062 +++++++++++++++++ .../multimodal_rag_langchain_infra.tf | 123 ++ examples/genai-rag-multimodal/outputs.tf | 40 + .../genai-rag-multimodal/terraform.tfvars | 5 + examples/genai-rag-multimodal/variables.tf | 82 ++ 6 files changed, 1451 insertions(+) create mode 100644 examples/genai-rag-multimodal/README.md create mode 100644 examples/genai-rag-multimodal/multimodal_rag_langchain.ipynb create mode 100644 examples/genai-rag-multimodal/multimodal_rag_langchain_infra.tf create mode 100644 examples/genai-rag-multimodal/outputs.tf create mode 100644 examples/genai-rag-multimodal/terraform.tfvars create mode 100644 examples/genai-rag-multimodal/variables.tf diff --git a/examples/genai-rag-multimodal/README.md b/examples/genai-rag-multimodal/README.md new file mode 100644 index 00000000..5f0c8421 --- /dev/null +++ b/examples/genai-rag-multimodal/README.md @@ -0,0 +1,139 @@ +# Multimodal RAG Langchain Example + +## Overview + +Retrieval Augmented Generation (RAG) has become a popular paradigm for enabling LLMs to access external data and also as a mechanism for [Grounding](https://cloud.google.com/vertex-ai/generative-ai/docs/grounding/overview), to mitigate against hallucinations. + +In this notebook, you will perform multimodal RAG by performing Q&A over a financial document filled with both text and images. + +This example is an adapted version of the sample Generative AI notebook from the Google Cloud codebase. You can find the original example and other notebooks in the [Google Cloud Platform Generative AI](https://github.com/GoogleCloudPlatform/generative-ai/tree/main) repository. + +The main modifications to the original example include: + +- Adaptations to comply with Cloud Foundation Toolkit security measures. +- Installation of additional libraries in the Conda environment. +- Use of Vertex AI Workbench to run the notebook with a custom Service Account. +- Implementation of Vector Search on Vertex AI with [Private Service Connect](https://cloud.google.com/vpc/docs/private-service-connect). + +## Requirements + +- Terraform v1.7.5 +- [Authenticated Google Cloud SDK 469.0.0](https://cloud.google.com/sdk/docs/authorizing) + +### Provision Infrastructure with Terraform + +- Update the `terraform.tfvars` file with values from your environment. + + ```terraform + kms_key = "projects/KMS-PROJECT-ID/locations/REGION/keyRings/ML-ENV-KEYRING/cryptoKeys/ML-ENV-KEY" + network = "projects/NETWORK-PROJECT-ID/global/networks/NETWORK-NAME" + subnet = "projects/NETWORK-PROJECT-ID/regions/REGION/subnetworks/SUBNET-NAME" + machine_learning_project = "MACHINE-LEARNING-PROJECT-ID" + vector_search_vpc_project = "NETWORK-PROJECT-ID" + ``` + +- Assuming you are deploying the example on top of the development environment, the following instructions will provide you more insight on how to retrieve these values: + - **NETWORK-PROJECT-ID**: Run `terraform output -raw restricted_host_project_id` on `gcp-networks` repository, inside the development environment directory and branch. + - **NETWORK-NAME**: Run `terraform output -raw restricted_network_name` on `gcp-networks` repository, inside the development environment directory and branch. + - **MACHINE-LEARNING-PROJECT-ID**: Run `terraform output -raw machine_learning_project_id` on `gcp-projects` repository, inside the Machine Learning business unit directory and on the development branch. + - **KMS-PROJECT-ID**, **ML-ENV-KEYRING**, **ML-ENV-KEY**: Run `terraform output machine_learning_kms_keys` on `gcp-projects` repository, inside the Machine Learning business unit directory and on the development branch. + - **REGION**: The chosen region. + +### Allow file download from Google Notebook Examples Bucket on VPC-SC Perimeter + +When running the Notebook, you will reach a step that downloads an example PDF file from a bucket, you need to add the egress rule below on the VPC-SC perimeter to allow the operation. + +```yaml +- egressFrom: + identities: + - serviceAccount:rag-notebook-runner@.iam.gserviceaccount.com + egressTo: + operations: + - methodSelectors: + - method: google.storage.buckets.list + - method: google.storage.buckets.get + - method: google.storage.objects.get + - method: google.storage.objects.list + serviceName: storage.googleapis.com + resources: + - projects/200612033880 # Google Cloud Example Project +``` + +## Usage + +Once all the requirements are set up, you can start by running and adjusting the notebook step-by-step. + +To run the notebook, open the Google Cloud Console on Vertex AI Workbench, open JupyterLab and upload the notebook (`multimodal_rag_langchain.ipynb`) to it. + +### Optional: Use `terraform output` and bash command to fill in fields in the notebook + +You can save some time adjusting the notebook by running the commands below: + +- Extract values from `terraform output` and validate. + + ```bash + export private_endpoint_ip_address=$(terraform output -raw private_endpoint_ip_address) + echo private_endpoint_ip_address=$private_endpoint_ip_address + + export host_vpc_project_id=$(terraform output -raw host_vpc_project_id) + echo host_vpc_project_id=$host_vpc_project_id + + export notebook_project_id=$(terraform output -raw notebook_project_id) + echo notebook_project_id=$notebook_project_id + + export vector_search_bucket_name=$(terraform output -raw vector_search_bucket_name) + echo vector_search_bucket_name=$vector_search_bucket_name + + export host_vpc_network=$(terraform output -raw host_vpc_network) + echo host_vpc_network=$host_vpc_network + ``` + +- Search and Replace using `sed` command. + + ```bash + sed -i "s//$private_endpoint_ip_address/g" multimodal_rag_langchain.ipynb + + sed -i "s//$host_vpc_project_id/g" multimodal_rag_langchain.ipynb + + sed -i "s//$notebook_project_id/g" multimodal_rag_langchain.ipynb + + sed -i "s//$vector_search_bucket_name/g" multimodal_rag_langchain.ipynb + + sed -i "s::$host_vpc_network:g" multimodal_rag_langchain.ipynb + ``` + +## Known Issues + +- `Error: Error creating Instance: googleapi: Error 400: value_to_check(https://compute.googleapis.com/compute/v1/projects/...) is not found`. + - When creating the VertexAI Workbench Instance through terraform you might face this issue. The issue is being tracked on this [link](https://github.com/hashicorp/terraform-provider-google/issues/17904). + - If you face this issue you will not be able to use terraform to create the instance, therefore, you will need to manually create it on [Google Cloud Console](https://console.cloud.google.com/vertex-ai/workbench/instances) using the same parameters. + + +## Inputs + +| Name | Description | Type | Default | Required | +|------|-------------|------|---------|:--------:| +| instance\_location | Vertex Workbench Instance Location | `string` | `"us-central1-a"` | no | +| kms\_key | The KMS key to use for disk encryption | `string` | n/a | yes | +| machine\_learning\_project | Machine Learning Project ID | `string` | n/a | yes | +| machine\_name | The name of the machine instance | `string` | `"rag-notebook-instance"` | no | +| machine\_type | The type of machine to use for the instance | `string` | `"e2-standard-2"` | no | +| network | The Host VPC network ID to connect the instance to | `string` | n/a | yes | +| service\_account\_name | The name of the service account | `string` | `"rag-notebook-runner"` | no | +| subnet | The subnet ID within the Host VPC network to use in Vertex Workbench and Private Service Connect | `string` | n/a | yes | +| vector\_search\_address\_name | The name of the address to create | `string` | `"vector-search-endpoint"` | no | +| vector\_search\_bucket\_location | Bucket Region | `string` | `"US-CENTRAL1"` | no | +| vector\_search\_ip\_region | The region to create the address in | `string` | `"us-central1"` | no | +| vector\_search\_vpc\_project | The project ID where the Host VPC network is located | `string` | n/a | yes | + +## Outputs + +| Name | Description | +|------|-------------| +| host\_vpc\_network | This is the Self-link of the Host VPC network | +| host\_vpc\_project\_id | This is the Project ID where the Host VPC network is located | +| notebook\_project\_id | The Project ID where the notebook will be run on | +| private\_endpoint\_ip\_address | The private IP address of the vector search endpoint | +| vector\_search\_bucket\_name | The name of the bucket that Vector Search will use | + + diff --git a/examples/genai-rag-multimodal/multimodal_rag_langchain.ipynb b/examples/genai-rag-multimodal/multimodal_rag_langchain.ipynb new file mode 100644 index 00000000..7e1cea33 --- /dev/null +++ b/examples/genai-rag-multimodal/multimodal_rag_langchain.ipynb @@ -0,0 +1,1062 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "ijGzTHJJUCPY" + }, + "outputs": [], + "source": [ + "# Copyright 2024 Google LLC\n", + "#\n", + "# Licensed under the Apache License, Version 2.0 (the \"License\");\n", + "# you may not use this file except in compliance with the License.\n", + "# You may obtain a copy of the License at\n", + "#\n", + "# https://www.apache.org/licenses/LICENSE-2.0\n", + "#\n", + "# Unless required by applicable law or agreed to in writing, software\n", + "# distributed under the License is distributed on an \"AS IS\" BASIS,\n", + "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n", + "# See the License for the specific language governing permissions and\n", + "# limitations under the License." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "NDsTUvKjwHBW" + }, + "source": [ + "# Multimodal Retrieval Augmented Generation (RAG) with Gemini, Vertex AI Vector Search, and LangChain\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " \"Google
Run in Colab Enterprise\n", + " \n", + " 		\n", + " \n", + " \"Google
Run in Colab\n", + " \n", + " 		\n", + " \n", + " \"Vertex
Open in Vertex AI Workbench\n", + " \n", + " 		\n", + " \n", + " \"GitHub
View on GitHub\n", + " \n", + "
" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "| | | \n", + "|-|-|\n", + "|Author(s) | [Holt Skinner](https://github.com/holtskinner) |" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VK1Q5ZYdVL4Y" + }, + "source": [ + "## Overview\n", + "\n", + "Retrieval augmented generation (RAG) has become a popular paradigm for enabling LLMs to access external data and also as a mechanism for grounding to mitigate against hallucinations.\n", + "\n", + "In this notebook, you will learn how to perform multimodal RAG where you will perform Q&A over a financial document filled with both text and images.\n", + "\n", + "### Gemini\n", + "\n", + "Gemini is a family of generative AI models developed by Google DeepMind that is designed for multimodal use cases. The Gemini API gives you access to the Gemini 1.0 Pro Vision and Gemini 1.0 Pro models.\n", + "\n", + "### Comparing text-based and multimodal RAG\n", + "\n", + "Multimodal RAG offers several advantages over text-based RAG:\n", + "\n", + "1. **Enhanced knowledge access:** Multimodal RAG can access and process both textual and visual information, providing a richer and more comprehensive knowledge base for the LLM.\n", + "2. **Improved reasoning capabilities:** By incorporating visual cues, multimodal RAG can make better informed inferences across different types of data modalities.\n", + "\n", + "This notebook shows you how to use RAG with Vertex AI Gemini API, and [multimodal embeddings](https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/multimodal-embeddings), to build a document search engine.\n", + "\n", + "Through hands-on examples, you will discover how to construct a multimedia-rich metadata repository of your document sources, enabling search, comparison, and reasoning across diverse information streams." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RQT500QqVPIb" + }, + "source": [ + "### Objectives\n", + "\n", + "This notebook provides a guide to building a document search engine using multimodal retrieval augmented generation (RAG), step by step:\n", + "\n", + "1. Extract and store metadata of documents containing both text and images, and generate embeddings the documents\n", + "2. Search the metadata with text queries to find similar text or images\n", + "3. Search the metadata with image queries to find similar images\n", + "4. Using a text query as input, search for contextual answers using both text and images" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KnpYxfesh2rI" + }, + "source": [ + "### Costs\n", + "\n", + "This tutorial uses billable components of Google Cloud:\n", + "\n", + "- Vertex AI\n", + "\n", + "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing) and use the [Pricing Calculator](https://cloud.google.com/products/calculator/) to generate a cost estimate based on your projected usage.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "DXJpXzKrh2rJ" + }, + "source": [ + "## Getting Started\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "N5afkyDMSBW5" + }, + "source": [ + "### Install Vertex AI SDK for Python and other dependencies\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 863 + }, + "executionInfo": { + "elapsed": 23606, + "status": "ok", + "timestamp": 1707899661283, + "user": { + "displayName": "", + "userId": "" + }, + "user_tz": -330 + }, + "id": "kc4WxYmLSBW5", + "outputId": "41191d43-b2b3-4bfd-c5e5-c0a53f03d1e2", + "tags": [] + }, + "outputs": [], + "source": [ + "%pip install -U -q google-cloud-aiplatform langchain-core langchain-google-vertexai==1.0.4 langchain-text-splitters langchain-experimental \"unstructured[all-docs]\" pypdf pydantic lxml pillow matplotlib opencv-python tiktoken" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%conda install -c conda-forge poppler tesseract -y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "R5Xep4W9lq-Z" + }, + "source": [ + "### Restart current runtime\n", + "\n", + "To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which will restart the current kernel." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Restart kernel after installs so that your environment can access the new packages\n", + "import IPython\n", + "\n", + "app = IPython.Application.instance()\n", + "app.kernel.do_shutdown(True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Create Private Endpoint IP Address\n", + "\n", + "Retrieve the IP address value created when setting up Private Service Connect for Vector Search. You can retrieve this value by running `terraform output` on the example." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "PRIVATE_ENDPOINT_IP_ADDRESS=\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Update Project Allowlist\n", + "\n", + "Insert the values of the Host Shared VPC Project and the current notebook project on the cell below.\n", + "\n", + "For example: `PROJECT_ALLOWLIST=[\"prj-d-shared-restricted-83dn\",\"prj-d-ml-machine-learning-0v09\"]` \n", + "\n", + "Remember that these values will be different on your environment" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "PROJECT_ALLOWLIST=[\"\", \"\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "SbmM4z7FOBpM" + }, + "source": [ + "
\n", + "⚠️ The kernel is going to restart. Please wait until it is finished before continuing to the next step. ⚠️\n", + "
\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O1vKZZoEh2rL" + }, + "source": [ + "### Define Google Cloud project information" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "executionInfo": { + "elapsed": 673, + "status": "ok", + "timestamp": 1707913890887, + "user": { + "displayName": "", + "userId": "" + }, + "user_tz": -330 + }, + "id": "gJqZ76rJh2rM", + "outputId": "ed164f41-97f3-411d-8cba-d853d71957b9", + "tags": [] + }, + "outputs": [], + "source": [ + "PROJECT_ID = \"\" # @param {type:\"string\"}\n", + "LOCATION = \"us-central1\" # @param {type:\"string\"}\n", + "\n", + "# For Vector Search Staging\n", + "GCS_BUCKET = \"\" # @param {type:\"string\"}\n", + "GCS_BUCKET_URI = f\"gs://{GCS_BUCKET}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Initialize the Vertex AI SDK" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "D48gUW5-h2rM", + "tags": [] + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform\n", + "\n", + "aiplatform.init(project=PROJECT_ID, location=LOCATION, staging_bucket=GCS_BUCKET_URI)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BuQwwRiniVFG" + }, + "source": [ + "### Import libraries\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "executionInfo": { + "elapsed": 647, + "status": "ok", + "timestamp": 1707913944466, + "user": { + "displayName": "", + "userId": "" + }, + "user_tz": -330 + }, + "id": "rtMowvm-yQ97", + "tags": [] + }, + "outputs": [], + "source": [ + "import base64\n", + "import os\n", + "import uuid\n", + "import re\n", + "\n", + "from typing import List, Tuple\n", + "\n", + "from IPython.display import display, Image, Markdown\n", + "\n", + "from langchain.prompts import PromptTemplate\n", + "from langchain.retrievers.multi_vector import MultiVectorRetriever\n", + "from langchain.storage import InMemoryStore\n", + "\n", + "from langchain_community.vectorstores import Chroma\n", + "\n", + "from langchain_core.documents import Document\n", + "from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n", + "from langchain_core.messages import AIMessage, HumanMessage\n", + "from langchain_core.output_parsers import StrOutputParser\n", + "\n", + "from langchain_text_splitters import CharacterTextSplitter\n", + "\n", + "from langchain_google_vertexai import (\n", + " VertexAI,\n", + " ChatVertexAI,\n", + " VertexAIEmbeddings,\n", + " VectorSearchVectorStore,\n", + ")\n", + "\n", + "from unstructured.partition.pdf import partition_pdf" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Data Loading" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "g7bKCQMFT7JT" + }, + "source": [ + "#### Get documents and images from GCS" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "executionInfo": { + "elapsed": 1757, + "status": "ok", + "timestamp": 1707913903524, + "user": { + "displayName": "", + "userId": "" + }, + "user_tz": -330 + }, + "id": "KwbL89zcY39N", + "outputId": "baa3a478-4d55-4b9c-e02f-6816a1d589b1", + "tags": [] + }, + "outputs": [], + "source": [ + "# Download documents and images used in this notebook\n", + "!gsutil -m rsync -r gs://github-repo/rag/intro_multimodal_rag/ .\n", + "print(\"Download completed\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ps1G-cCfpibN" + }, + "source": [ + "## Partition PDF tables, text, and images" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "jqLsy3iZ5t-R" + }, + "source": [ + "### The data\n", + "\n", + "The source data that you will use in this notebook is a modified version of [Google-10K](https://abc.xyz/assets/investor/static/pdf/20220202_alphabet_10K.pdf) which provides a comprehensive overview of the company's financial performance, business operations, management, and risk factors. As the original document is rather large, you will be using [a modified version with only 14 pages](https://storage.googleapis.com/github-repo/rag/multimodal_rag_langchain/google-10k-sample-14pages.pdf) instead. Although it's truncated, the sample document still contains text along with images such as tables, charts, and graphs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "pdf_folder_path = \"data/\"\n", + "pdf_file_name = \"intro_multimodal_rag_old_version/data/google-10k-sample-part1.pdf\"\n", + "\n", + "# Extract images, tables, and chunk text from a PDF file.\n", + "raw_pdf_elements = partition_pdf(\n", + " filename=pdf_file_name,\n", + " extract_images_in_pdf=False,\n", + " infer_table_structure=True,\n", + " chunking_strategy=\"by_title\",\n", + " max_characters=4000,\n", + " new_after_n_chars=3800,\n", + " combine_text_under_n_chars=2000,\n", + " image_output_dir_path=pdf_folder_path,\n", + ")\n", + "\n", + "# Categorize extracted elements from a PDF into tables and texts.\n", + "tables = []\n", + "texts = []\n", + "for element in raw_pdf_elements:\n", + " if \"unstructured.documents.elements.Table\" in str(type(element)):\n", + " tables.append(str(element))\n", + " elif \"unstructured.documents.elements.CompositeElement\" in str(type(element)):\n", + " texts.append(str(element))\n", + "\n", + "# Optional: Enforce a specific token size for texts\n", + "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n", + " chunk_size=10000, chunk_overlap=0\n", + ")\n", + "joined_texts = \" \".join(texts)\n", + "texts_4k_token = text_splitter.split_text(joined_texts)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "MODEL_NAME = \"gemini-1.0-pro-vision\"\n", + "\n", + "\n", + "# Generate summaries of text elements\n", + "def generate_text_summaries(\n", + " texts: List[str], tables: List[str], summarize_texts: bool = False\n", + ") -> Tuple[List, List]:\n", + " \"\"\"\n", + " Summarize text elements\n", + " texts: List of str\n", + " tables: List of str\n", + " summarize_texts: Bool to summarize texts\n", + " \"\"\"\n", + "\n", + " # Prompt\n", + " prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n", + " These summaries will be embedded and used to retrieve the raw text or table elements. \\\n", + " Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} \"\"\"\n", + " prompt = PromptTemplate.from_template(prompt_text)\n", + " empty_response = RunnableLambda(\n", + " lambda x: AIMessage(content=\"Error processing document\")\n", + " )\n", + " # Text summary chain\n", + " model = VertexAI(\n", + " temperature=0, model_name=MODEL_NAME, max_output_tokens=1024\n", + " ).with_fallbacks([empty_response])\n", + " summarize_chain = {\"element\": lambda x: x} | prompt | model | StrOutputParser()\n", + "\n", + " # Initialize empty summaries\n", + " text_summaries = []\n", + " table_summaries = []\n", + "\n", + " # Apply to text if texts are provided and summarization is requested\n", + " if texts:\n", + " if summarize_texts:\n", + " text_summaries = summarize_chain.batch(texts, {\"max_concurrency\": 1})\n", + " else:\n", + " text_summaries = texts\n", + "\n", + " # Apply to tables if tables are provided\n", + " if tables:\n", + " table_summaries = summarize_chain.batch(tables, {\"max_concurrency\": 1})\n", + "\n", + " return text_summaries, table_summaries\n", + "\n", + "\n", + "# Get text, table summaries\n", + "text_summaries, table_summaries = generate_text_summaries(\n", + " texts_4k_token, tables, summarize_texts=True\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def encode_image(image_path):\n", + " \"\"\"Getting the base64 string\"\"\"\n", + " with open(image_path, \"rb\") as image_file:\n", + " return base64.b64encode(image_file.read()).decode(\"utf-8\")\n", + "\n", + "\n", + "def image_summarize(img_base64, prompt):\n", + " \"\"\"Make image summary\"\"\"\n", + " model = ChatVertexAI(model_name=\"gemini-pro-vision\", max_output_tokens=1024)\n", + "\n", + " msg = model(\n", + " [\n", + " HumanMessage(\n", + " content=[\n", + " {\"type\": \"text\", \"text\": prompt},\n", + " {\n", + " \"type\": \"image_url\",\n", + " \"image_url\": {\"url\": f\"data:image/png;base64,{img_base64}\"},\n", + " },\n", + " ]\n", + " )\n", + " ]\n", + " )\n", + " return msg.content\n", + "\n", + "\n", + "def generate_img_summaries(path):\n", + " \"\"\"\n", + " Generate summaries and base64 encoded strings for images\n", + " path: Path to list of .jpg files extracted by Unstructured\n", + " \"\"\"\n", + "\n", + " # Store base64 encoded images\n", + " img_base64_list = []\n", + "\n", + " # Store image summaries\n", + " image_summaries = []\n", + "\n", + " # Prompt\n", + " prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\\n", + " These summaries will be embedded and used to retrieve the raw image. \\\n", + " Give a concise summary of the image that is well optimized for retrieval.\n", + " If it's a table, extract all elements of the table.\n", + " If it's a graph, explain the findings in the graph.\n", + " Do not include any numbers that are not mentioned in the image.\n", + " \"\"\"\n", + "\n", + " # Apply to images\n", + " for img_file in sorted(os.listdir(path)):\n", + " if img_file.endswith(\".png\"):\n", + " img_path = os.path.join(path, img_file)\n", + " base64_image = encode_image(img_path)\n", + " img_base64_list.append(base64_image)\n", + " image_summaries.append(image_summarize(base64_image, prompt))\n", + "\n", + " return img_base64_list, image_summaries\n", + "\n", + "\n", + "# Image summaries\n", + "img_base64_list, image_summaries = generate_img_summaries(\".\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create & Deploy Vertex AI Vector Search Index & Endpoint\n", + "\n", + "Skip this step if you already have Vector Search set up.\n", + "\n", + "- https://console.cloud.google.com/vertex-ai/matching-engine/indexes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Create [`MatchingEngineIndex`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.MatchingEngineIndex)\n", + " - https://cloud.google.com/vertex-ai/docs/vector-search/create-manage-index" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings\n", + "DIMENSIONS = 768 # Dimensions output from textembedding-gecko\n", + "\n", + "index = aiplatform.MatchingEngineIndex.create_tree_ah_index(\n", + " display_name=\"mm_rag_langchain_index\",\n", + " dimensions=DIMENSIONS,\n", + " approximate_neighbors_count=150,\n", + " leaf_node_embedding_count=500,\n", + " leaf_nodes_to_search_percent=7,\n", + " description=\"Multimodal RAG LangChain Index\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Create [`MatchingEngineIndexEndpoint`](https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.MatchingEngineIndexEndpoint)\n", + " - https://cloud.google.com/vertex-ai/docs/vector-search/deploy-index-public" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "DEPLOYED_INDEX_ID = \"mm_rag_langchain_index_endpoint\"\n", + "\n", + "index_endpoint = aiplatform.MatchingEngineIndexEndpoint.create(\n", + " display_name=DEPLOYED_INDEX_ID,\n", + " description=\"Multimodal RAG LangChain Index Endpoint\",\n", + " enable_private_service_connect=True,\n", + " project_allowlist=PROJECT_ALLOWLIST,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Deploy Index to Index Endpoint\n", + " - NOTE: This will take a while to run.\n", + " - You can stop this cell after starting it instead of waiting for deployment.\n", + " - You can check the status at https://console.cloud.google.com/vertex-ai/matching-engine/indexes" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "index_endpoint = index_endpoint.deploy_index(\n", + " index=index, deployed_index_id=\"mm_rag_langchain_deployed_index\"\n", + ")\n", + "index_endpoint.deployed_indexes" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* You will need to create the service attachment to the Private Service Connect IP" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "SERVICE_ATTACHMENT=index_endpoint.deployed_indexes[0].private_endpoints.service_attachment\n", + "\n", + "SERVICE_ATTACHMENT" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* Retrieve the value of the service attachment and execute the command below on your local machine:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "NETWORK=\"\"\n", + "NETWORK_PROJECT_ID=\"\"\n", + "\n", + "!gcloud compute forwarding-rules create vector-search-endpoint \\\n", + " --network={NETWORK} \\\n", + " --address=vector-search-endpoint \\\n", + " --target-service-attachment={SERVICE_ATTACHMENT} \\\n", + " --project={NETWORK_PROJECT_ID} \\\n", + " --region=us-central1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create retriever & load documents" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Create [`VectorSearchVectorStore`](https://api.python.langchain.com/en/latest/vectorstores/langchain_google_vertexai.vectorstores.vectorstores.VectorSearchVectorStore.html) with Vector Search Index ID and Endpoint ID.\n", + "- Use [`textembedding-gecko`](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings) as embedding model." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# The vectorstore to use to index the summaries\n", + "vectorstore = VectorSearchVectorStore.from_components(\n", + " project_id=PROJECT_ID,\n", + " region=LOCATION,\n", + " gcs_bucket_name=GCS_BUCKET,\n", + " index_id=index.name,\n", + " endpoint_id=index_endpoint.name,\n", + " embedding=VertexAIEmbeddings(model_name=\"textembedding-gecko@003\"),\n", + " private_service_connect_ip_address=PRIVATE_ENDPOINT_IP_ADDRESS,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Create Multi-Vector Retriever using the vector store you created.\n", + "- Since vector stores only contain the embedding and an ID, you'll also need to create a document store indexed by ID to get the original source documents after searching for embeddings." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "docstore = InMemoryStore()\n", + "\n", + "id_key = \"doc_id\"\n", + "# Create the multi-vector retriever\n", + "retriever_multi_vector_img = MultiVectorRetriever(\n", + " vectorstore=vectorstore,\n", + " docstore=docstore,\n", + " id_key=id_key,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* Set Private Service Connect address" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "retriever_multi_vector_img.vectorstore._searcher._endpoint.private_service_connect_ip_address = PRIVATE_ENDPOINT_IP_ADDRESS" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Load data into Document Store and Vector Store" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# Raw Document Contents\n", + "doc_contents = texts + tables + img_base64_list\n", + "\n", + "doc_ids = [str(uuid.uuid4()) for _ in doc_contents]\n", + "summary_docs = [\n", + " Document(page_content=s, metadata={id_key: doc_ids[i]})\n", + " for i, s in enumerate(text_summaries + table_summaries + image_summaries)\n", + "]\n", + "\n", + "retriever_multi_vector_img.docstore.mset(list(zip(doc_ids, doc_contents)))\n", + "\n", + "# If using Vertex AI Vector Search, this will take a while to complete.\n", + "# You can cancel this cell and continue later.\n", + "retriever_multi_vector_img.vectorstore.add_documents(summary_docs)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create Chain with Retriever and Gemini LLM" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def looks_like_base64(sb):\n", + " \"\"\"Check if the string looks like base64\"\"\"\n", + " return re.match(\"^[A-Za-z0-9+/]+[=]{0,2}$\", sb) is not None\n", + "\n", + "\n", + "def is_image_data(b64data):\n", + " \"\"\"\n", + " Check if the base64 data is an image by looking at the start of the data\n", + " \"\"\"\n", + " image_signatures = {\n", + " b\"\\xFF\\xD8\\xFF\": \"jpg\",\n", + " b\"\\x89\\x50\\x4E\\x47\\x0D\\x0A\\x1A\\x0A\": \"png\",\n", + " b\"\\x47\\x49\\x46\\x38\": \"gif\",\n", + " b\"\\x52\\x49\\x46\\x46\": \"webp\",\n", + " }\n", + " try:\n", + " header = base64.b64decode(b64data)[:8] # Decode and get the first 8 bytes\n", + " for sig, format in image_signatures.items():\n", + " if header.startswith(sig):\n", + " return True\n", + " return False\n", + " except Exception:\n", + " return False\n", + "\n", + "\n", + "def split_image_text_types(docs):\n", + " \"\"\"\n", + " Split base64-encoded images and texts\n", + " \"\"\"\n", + " b64_images = []\n", + " texts = []\n", + " for doc in docs:\n", + " # Check if the document is of type Document and extract page_content if so\n", + " if isinstance(doc, Document):\n", + " doc = doc.page_content\n", + " if looks_like_base64(doc) and is_image_data(doc):\n", + " b64_images.append(doc)\n", + " else:\n", + " texts.append(doc)\n", + " return {\"images\": b64_images, \"texts\": texts}\n", + "\n", + "\n", + "def img_prompt_func(data_dict):\n", + " \"\"\"\n", + " Join the context into a single string\n", + " \"\"\"\n", + " formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n", + " messages = [\n", + " {\n", + " \"type\": \"text\",\n", + " \"text\": (\n", + " \"You are financial analyst tasking with providing investment advice.\\n\"\n", + " \"You will be given a mix of text, tables, and image(s) usually of charts or graphs.\\n\"\n", + " \"Use this information to provide investment advice related to the user's question. \\n\"\n", + " f\"User-provided question: {data_dict['question']}\\n\\n\"\n", + " \"Text and / or tables:\\n\"\n", + " f\"{formatted_texts}\"\n", + " ),\n", + " }\n", + " ]\n", + "\n", + " # Adding image(s) to the messages if present\n", + " if data_dict[\"context\"][\"images\"]:\n", + " for image in data_dict[\"context\"][\"images\"]:\n", + " messages.append(\n", + " {\n", + " \"type\": \"image_url\",\n", + " \"image_url\": {\"url\": f\"data:image/jpeg;base64,{image}\"},\n", + " }\n", + " )\n", + " return [HumanMessage(content=messages)]\n", + "\n", + "\n", + "# Create RAG chain\n", + "chain_multimodal_rag = (\n", + " {\n", + " \"context\": retriever_multi_vector_img | RunnableLambda(split_image_text_types),\n", + " \"question\": RunnablePassthrough(),\n", + " }\n", + " | RunnableLambda(img_prompt_func)\n", + " | ChatVertexAI(\n", + " temperature=0, model_name=\"gemini-pro-vision\", max_output_tokens=1024\n", + " ) # Multi-modal LLM\n", + " | StrOutputParser()\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Process user query" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"\"\"\n", + " - What are the critical difference between various graphs for Class A Share?\n", + " - Which index best matches Class A share performance closely where Google is not already a part? Explain the reasoning.\n", + " - Identify key chart patterns for Google Class A shares.\n", + " - What is cost of revenues, operating expenses and net income for 2020. Do mention the percentage change\n", + " - What was the effect of Covid in the 2020 financial year?\n", + " - What are the total revenues for APAC and USA for 2021?\n", + " - What is deferred income taxes?\n", + " - How do you compute net income per share?\n", + " - What drove percentage change in the consolidated revenue and cost of revenue for the year 2021 and was there any effect of Covid?\n", + " - What is the cause of 41% increase in revenue from 2020 to 2021 and how much is dollar change?\n", + "\"\"\"" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Get Retrieved documents" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# List of source documents\n", + "docs = retriever_multi_vector_img.get_relevant_documents(query, limit=10)\n", + "\n", + "source_docs = split_image_text_types(docs)\n", + "\n", + "print(source_docs[\"texts\"])\n", + "\n", + "for i in source_docs[\"images\"]:\n", + " display(Image(base64.b64decode(i)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Get generative response" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "result = chain_multimodal_rag.invoke(query)\n", + "\n", + "Markdown(result)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KwNrHCqbi3xi" + }, + "source": [ + "## Conclusions" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "05jynhZnkgxn" + }, + "source": [ + "Congratulations on making it through this multimodal RAG notebook!\n", + "\n", + "While multimodal RAG can be quite powerful, note that it can face some limitations:\n", + "\n", + "* **Data dependency:** Needs high-accuracy data from the text and visuals.\n", + "* **Computationally demanding:** Generating embeddings from multimodal data is resource-intensive.\n", + "* **Domain specific:** Models trained on general data may not shine in specialized fields like medicine.\n", + "* **Black box:** Understanding how these models work can be tricky, hindering trust and adoption.\n", + "\n", + "\n", + "Despite these challenges, multimodal RAG represents a significant step towards search and retrieval systems that can handle diverse, multimodal data." + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "environment": { + "kernel": "python3", + "name": "common-cpu.m116", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cpu:m116" + }, + "kernelspec": { + "display_name": "Python 3 (Local)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.2" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +} diff --git a/examples/genai-rag-multimodal/multimodal_rag_langchain_infra.tf b/examples/genai-rag-multimodal/multimodal_rag_langchain_infra.tf new file mode 100644 index 00000000..6b5e7871 --- /dev/null +++ b/examples/genai-rag-multimodal/multimodal_rag_langchain_infra.tf @@ -0,0 +1,123 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +data "google_project" "ml_project" { + project_id = var.machine_learning_project +} + +resource "google_workbench_instance" "instance" { + disable_proxy_access = false + instance_owners = [] + location = var.instance_location + name = var.machine_name + project = var.machine_learning_project + + gce_setup { + service_accounts { + email = google_service_account.notebook_runner.email + } + disable_public_ip = true + machine_type = var.machine_type + metadata = { + "disable-mixer" = "false" + "notebook-disable-downloads" = "true" + "notebook-disable-root" = "true" + "notebook-disable-terminal" = "true" + "notebook-upgrade-schedule" = "00 19 * * MON" + "report-dns-resolution" = "true" + "report-event-health" = "true" + "terraform" = "true" + } + tags = [ + "egress-internet", + ] + boot_disk { + disk_encryption = "CMEK" + disk_size_gb = "150" + disk_type = "PD_SSD" + kms_key = var.kms_key + } + data_disks { + disk_encryption = "CMEK" + disk_size_gb = "150" + disk_type = "PD_SSD" + kms_key = var.kms_key + } + network_interfaces { + network = var.network + subnet = var.subnet + } + vm_image { + family = "workbench-instances" + project = "cloud-notebooks-managed" + } + } +} + +resource "random_string" "suffix" { + length = 10 + special = false + upper = false +} + +resource "google_storage_bucket" "vector_search_bucket" { + name = "vector-search-${random_string.suffix.result}" + location = var.vector_search_bucket_location + storage_class = "REGIONAL" + project = var.machine_learning_project + uniform_bucket_level_access = true +} + +resource "google_compute_address" "vector_search_static_ip" { + name = var.vector_search_address_name + region = var.vector_search_ip_region + subnetwork = var.subnet + project = var.vector_search_vpc_project + address_type = "INTERNAL" +} + +resource "google_service_account" "notebook_runner" { + account_id = var.service_account_name + display_name = "RAG Notebook Runner Service Account" + project = var.machine_learning_project +} + +resource "google_project_iam_member" "notebook_runner_roles" { + for_each = toset([ + "roles/aiplatform.user" + ]) + project = var.machine_learning_project + role = each.key + member = google_service_account.notebook_runner.member +} + +resource "google_storage_bucket_iam_member" "notebook_runner_bucket_admin" { + bucket = google_storage_bucket.vector_search_bucket.name + role = "roles/storage.admin" + member = google_service_account.notebook_runner.member +} + +# Service Agent Role Assignment - Allows creation of workbench instance when using var.kms_key + +resource "google_kms_crypto_key_iam_member" "service_agent_kms_key_binding" { + for_each = toset([ + "serviceAccount:service-${data.google_project.ml_project.number}@compute-system.iam.gserviceaccount.com", + "serviceAccount:service-${data.google_project.ml_project.number}@gcp-sa-notebooks.iam.gserviceaccount.com" + ]) + crypto_key_id = var.kms_key + role = "roles/cloudkms.cryptoKeyEncrypterDecrypter" + member = each.value +} diff --git a/examples/genai-rag-multimodal/outputs.tf b/examples/genai-rag-multimodal/outputs.tf new file mode 100644 index 00000000..7ad847ca --- /dev/null +++ b/examples/genai-rag-multimodal/outputs.tf @@ -0,0 +1,40 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "private_endpoint_ip_address" { + description = "The private IP address of the vector search endpoint" + value = google_compute_address.vector_search_static_ip.address +} + +output "host_vpc_project_id" { + description = "This is the Project ID where the Host VPC network is located" + value = google_compute_address.vector_search_static_ip.project +} + +output "host_vpc_network" { + description = "This is the Self-link of the Host VPC network" + value = var.network +} + +output "notebook_project_id" { + description = "The Project ID where the notebook will be run on" + value = var.machine_learning_project +} + +output "vector_search_bucket_name" { + description = "The name of the bucket that Vector Search will use" + value = google_storage_bucket.vector_search_bucket.name +} diff --git a/examples/genai-rag-multimodal/terraform.tfvars b/examples/genai-rag-multimodal/terraform.tfvars new file mode 100644 index 00000000..34944ca3 --- /dev/null +++ b/examples/genai-rag-multimodal/terraform.tfvars @@ -0,0 +1,5 @@ +kms_key = +network = +subnet = +machine_learning_project = +vector_search_vpc_project = diff --git a/examples/genai-rag-multimodal/variables.tf b/examples/genai-rag-multimodal/variables.tf new file mode 100644 index 00000000..8c3ff9da --- /dev/null +++ b/examples/genai-rag-multimodal/variables.tf @@ -0,0 +1,82 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "service_account_name" { + description = "The name of the service account" + type = string + default = "rag-notebook-runner" +} + +variable "vector_search_bucket_location" { + description = "Bucket Region" + type = string + default = "US-CENTRAL1" +} + +variable "machine_learning_project" { + description = "Machine Learning Project ID" + type = string +} + +variable "vector_search_address_name" { + description = "The name of the address to create" + type = string + default = "vector-search-endpoint" +} + +variable "vector_search_ip_region" { + description = "The region to create the address in" + type = string + default = "us-central1" +} + +variable "vector_search_vpc_project" { + description = "The project ID where the Host VPC network is located" + type = string +} + +variable "kms_key" { + description = "The KMS key to use for disk encryption" + type = string +} + +variable "network" { + description = "The Host VPC network ID to connect the instance to" + type = string +} + +variable "subnet" { + description = "The subnet ID within the Host VPC network to use in Vertex Workbench and Private Service Connect" + type = string +} + +variable "machine_type" { + description = "The type of machine to use for the instance" + type = string + default = "e2-standard-2" +} + +variable "machine_name" { + description = "The name of the machine instance" + type = string + default = "rag-notebook-instance" +} + +variable "instance_location" { + description = "Vertex Workbench Instance Location" + type = string + default = "us-central1-a" +}