feat(docs): Add 5-appinfra instructions to deploy on foundation docs (#…

…49)

* adding 5-appinfra

* add key

* update

* update service catalog files

* add more details

* update docs

* add more info

* remove trailing whitespace

README.md

@@ -2,36 +2,44 @@
 
 ## Overview
 
-This repository serves as a example for configuring an environment for the development and deployment of Machine Learning applications using the Vertex AI platform on Google Cloud. It seamlessly integrates the Cloud Foundation Toolkit (CFT) and implements robust security measures, drawing heavily from the [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/tree/v4.0.0) codebase.
-
-The repo is separated in distinct Terraform projects, each within their own directory that must be applied separately, but in sequence, for more information about each step, please refer to [terraform-google-enterprise-genai](https://github.com/terraform-google-modules/terraform-google-enterprise-genai/tree/v4.0.0). Comparing to the foundation repository, the key differences from the steps in foundation to steps in these repository are:
-
-Alternatively, the user can follow steps on [`docs/deploy_on_foundation_v4.0.0.md`](./docs/deploy_on_foundation_v4.0.0.md), to deploy the codebase on top of an existing foundation instance.
-
-* [1. org](./1-org/)
-    * Specific to this repository, it will also configure Machine Learning Organization Policies.
-* [2. environments](./2-environments/)
-    * This repository will also establish organization and environment-level Cloud Key Management Service (KMS) keyrings during this stage.
-* [3. networks-dual-svpc](./3-networks-dual-svpc/)
-    * On this repository, it will also configure a private DNS zone for workbench instances to use either `private.googleapis.com` or `restricted.googleapis.com`.
-* [4. projects](./4-projects/)
-    * Instead of creating `business_unit_1` and `business_unit_2`, this repository exclusively creates `business_unit_3`.
-    * Additionally, it will establish a Service Catalog project capable of hosting terraform solutions and an artifacts project.
-    * Will create a Machine Learning project for each environment.
-* [5. app-infra](./5-app-infra/)
-    * The purpose of this step is to execute a series of steps necessary to deploy and run a Machine Learning Application.
-
-Additional steps were added to provide an example Machine Learning application:
-
-* [machine-learning-pipeline](./examples/machine-learning-pipeline)
-    * This additional step, adds an interactive coding and experimentation, deploying the Vertex Workbench for data scientists.
-    * The step will guide you through creating a ML pipeline using a notebook on Google Vertex AI Workbench Instance.
-    * After promoting the ML pipeline, it is triggered by Cloud Build upon staging branch merges, trains and deploys a model using the census income dataset.
-    * Model deployment and monitoring occur in the prod environment.
-    * Following successful pipeline runs, a new model version is deployed for A/B testing.
-
-* [7. composer](./7-composer/)
-    * Used for code reference.
-
-* [7. vertexpipeline](./7-vertexpipeline/)
-    * Used for code reference, will be used for creating the Machine Learning pipeline.
+This repository serves as a example for configuring an environment for the development and deployment of Machine Learning applications using the Vertex AI platform on Google Cloud. It seamlessly integrates the Cloud Foundation Toolkit (CFT) and implements robust security measures, drawing heavily from the [terraform-example-foundation v4.0.0](https://github.com/terraform-google-modules/terraform-example-foundation/tree/v4.0.0) codebase.
+
+The repository is divided into distinct Terraform projects, each located in its own directory. These projects must be applied separately but in sequence. For detailed information about each step, please refer to [terraform-example-foundation v4.0.0](https://github.com/terraform-google-modules/terraform-example-foundation/tree/v4.0.0). The user has two options when deploying this codebase:
+
+- Following the individual project steps as outlined in this repository, under `0-bootstrap` to `5-appinfra` directories.
+- Deploy the codebase on top of an existing Enterprise Foundations Blueprint instance by following the steps detailed in [`docs/deploy_on_foundation_v4.0.0.md`](./docs/deploy_on_foundation_v4.0.0.md).
+  > NOTE: If the user currently does not have a Enterprise Foundations Blueprint deployed, he can follow the steps outlined in [terraform-example-foundation v4.0.0](https://github.com/terraform-google-modules/terraform-example-foundation/tree/v4.0.0) to deploy it.
+
+## Main Modifications made to Enterprise Foundations Blueprint
+
+- [1. org](./1-org/)
+  - Specific to this repository, it will also configure Machine Learning Organization Policies.
+  - Create Organization Level Keyring.
+- [2. environments](./2-environments/)
+  - This repository will also establish organization and environment-level Cloud Key Management Service (KMS) keyrings during this stage.
+  - Create support for environment-level logging.
+- [3. networks-dual-svpc](./3-networks-dual-svpc/)
+  - On this repository, it will also configure a private DNS zone for workbench instances to use either `private.googleapis.com` or `restricted.googleapis.com`.
+  - Custom Firewall Rules (`allow_all_ingress_ranges` and `allow_all_egress_ranges`).
+  - Enable Cloud NAT.
+  - Attach Environment-level Logging Project and Environment-level KMS Project to VPC-SC Perimeter.
+- [4. projects](./4-projects/)
+  - Instead of creating `business_unit_1` and `business_unit_2`, this repository will create `ml_business_unit`.
+  - Additionally, it will establish a Service Catalog project capable of hosting terraform solutions and an Artifacts project, both under the `common` folder.
+  - Will create a Machine Learning project for each environment, that is inside a VPC-SC Perimeter and can be used for deploying Machine Learning Workloads.
+- [5. app-infra](./5-app-infra/)
+  - Deploys a Service Catalog Pipeline, that can be used for packaging terraform modules.
+  - Creates an Artifacts Pipeline, that can be used to create organization-wide custom docker images.
+
+## Examples
+
+- [genai-rag-multimodal](./examples/genai-rag-multimodal)
+  - Multimodal RAG by performing Q&A over a financial document filled with both text and images.
+  - Use RAGAS for RAG chain evaluation.
+
+- [machine-learning-pipeline](./examples/machine-learning-pipeline)
+  - This example, adds an interactive coding and experimentation, deploying the Vertex Workbench for data scientists.
+  - The step will guide you through creating a ML pipeline using a notebook on Google Vertex AI Workbench Instance.
+  - After promoting the ML pipeline, it is triggered by Cloud Build upon staging branch merges, trains and deploys a model using the census income dataset.
+  - Model deployment and monitoring occur in the prod environment.
+  - Following successful pipeline runs, a new model version is deployed for A/B testing.

docs/assets/terraform/4-projects/ml_business_unit/shared/outputs.tf

@@ -92,3 +92,8 @@ output "artifacts_repo_id" {
   description = "ID of the Artifacts repository"
   value       = module.ml_infra_projects.artifacts_repo_id
 }
+
+output "shared_level_keyrings" {
+  description = "Keyrings used on shared level project creation"
+  value       = local.shared_kms_key_ring
+}

docs/assets/terraform/4-projects/shared.auto.example.tfvars

@@ -17,6 +17,6 @@
 # We suggest you to use the same region from the 0-bootstrap step
 default_region = "us-central1"
 
-cloud_source_service_catalog_repo_name = "svc-catalog"
+cloud_source_service_catalog_repo_name = "service-catalog"
 
 cloud_source_artifacts_repo_name = "publish-artifacts"

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/common.auto.example.tfvars

@@ -0,0 +1,19 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+instance_region = "us-central1" // should be one of the regions used to create network on step 3-networks
+
+remote_state_bucket = "REMOTE_STATE_BUCKET"

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/README.md

@@ -0,0 +1,15 @@
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| instance\_region | The region where compute instance will be created. A subnetwork must exists in the instance region. | `string` | n/a | yes |
+| remote\_state\_bucket | Backend bucket to load remote state information from previous steps. | `string` | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| cloudbuild\_trigger\_id | n/a |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/backend.tf

@@ -0,0 +1,22 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+terraform {
+  backend "gcs" {
+    bucket = "UPDATE_APP_INFRA_BUCKET"
+    prefix = "terraform/app-infra/ml_business_unit/shared"
+  }
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/common.auto.tfvars

@@ -0,0 +1 @@
+../../common.auto.tfvars

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/locals.tf

@@ -0,0 +1,20 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  business_unit = "ml_business_unit"
+  environment   = "common"
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/outputs.tf

@@ -0,0 +1,34 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# output "trigger_sa_account_id" {
+#   description = "Account id of service account cloudbuild."
+#   value       = module.artifact_pipeline.trigger_sa_account_id
+# }
+
+# output "cloudbuild_v2_repo_id" {
+#   description = "Repository ID of cloudbuild repository"
+#   value       = module.artifact_pipeline.cloudbuild_v2_repo_id
+# }
+
+# output "kms_key_id" {
+#   description = "Projects Key ID for encrytion"
+#   value       = module.artifact_pipeline.kms_key_id
+# }
+
+output "cloudbuild_trigger_id" {
+  value = module.artifact_publish.cloudbuild_trigger_id
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/publish_artifacts.tf

@@ -0,0 +1,47 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  region_kms_keyring = [for i in local.shared_keyrings : i if split("/", i)[3] == var.instance_region]
+}
+
+data "google_project" "common_artifacts" {
+  project_id = local.common_artifacts_project_id
+}
+
+module "artifact_publish" {
+  source = "../../modules/publish_artifacts"
+
+  environment = local.environment
+  description = "Publish Artifacts for ML Projects"
+  project_id  = local.common_artifacts_project_id
+  name        = local.artifacts_repo_name
+  format      = "DOCKER"
+  region      = var.instance_region
+  cleanup_policies = [{
+    id     = "keep-tagged-release"
+    action = "KEEP"
+    condition = [
+      {
+        tag_state             = "TAGGED",
+        tag_prefixes          = ["release"],
+        package_name_prefixes = ["webapp", "mobile"]
+      }
+    ]
+  }]
+
+  kms_crypto_key = "${one(local.region_kms_keyring)}/cryptoKeys/${data.google_project.common_artifacts.name}"
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/remote.tf

@@ -0,0 +1,31 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+locals {
+  common_artifacts_project_id = data.terraform_remote_state.projects_shared.outputs.common_artifacts_project_id
+  service_catalog_project_id  = data.terraform_remote_state.projects_shared.outputs.service_catalog_project_id
+  artifacts_repo_name         = data.terraform_remote_state.projects_shared.outputs.artifacts_repo_name
+  shared_keyrings             = data.terraform_remote_state.projects_shared.outputs.shared_level_keyrings
+}
+
+data "terraform_remote_state" "projects_shared" {
+  backend = "gcs"
+
+  config = {
+    bucket = var.remote_state_bucket
+    prefix = "terraform/projects/${local.business_unit}/shared"
+  }
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/variables.tf

@@ -0,0 +1,25 @@
+/**
+ * Copyright 2021 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+variable "instance_region" {
+  description = "The region where compute instance will be created. A subnetwork must exists in the instance region."
+  type        = string
+}
+
+variable "remote_state_bucket" {
+  description = "Backend bucket to load remote state information from previous steps."
+  type        = string
+}

docs/assets/terraform/5-appinfra/artifact-publish-infra-repo/ml_business_unit/shared/versions.tf

@@ -0,0 +1,43 @@
+/**
+ * Copyright 2022 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+terraform {
+  required_version = ">= 0.13"
+
+  required_providers {
+
+    google = {
+      source  = "hashicorp/google"
+      version = ">= 3.77, < 6"
+    }
+
+    google-beta = {
+      source  = "hashicorp/google-beta"
+      version = ">= 3.77, < 6"
+    }
+
+    null = {
+      source  = "hashicorp/null"
+      version = "~> 3.0"
+    }
+
+    random = {
+      source  = "hashicorp/random"
+      version = "~> 3.1"
+    }
+
+  }
+}

docs/assets/terraform/5-appinfra/artifacts-pipeline/README.md

@@ -0,0 +1,2 @@
+# ml-foundations-docker
+Dockerfile repository for ml-foundations artifacts project

docs/assets/terraform/5-appinfra/artifacts-pipeline/images/tf2-cpu.2-13:0.1/Dockerfile

@@ -0,0 +1,17 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+FROM us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-13:latest
+RUN echo "Hello World" > helloworld.txt
+