diff --git a/anthos-onprem-terraform/README.md b/anthos-onprem-terraform/README.md index 6fd03a16..e8a8e4ca 100644 --- a/anthos-onprem-terraform/README.md +++ b/anthos-onprem-terraform/README.md @@ -11,9 +11,10 @@ For more information, see the reference documentation for each resource. #### Anthos clusters on bare metal (ABM) -| Type | Sample _(by loadbalancer type)_ | Terraform resources | -| ---------------- | ---------------------------------------------- | ------------------- | -| **user** cluster | Bundled [MetalLB](./abm_user_cluster_metallb/) | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster)
[google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | +| Type | Sample _(by loadbalancer type)_ | Terraform resources | +| ---------------- | ---------------------------------------------- | ------------------- | +| **user** cluster | Bundled [MetalLB](./abm_user_cluster_metallb/) | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster)
[google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | +| **user** cluster | [ManualLB](./abm_user_cluster_manuallb/) | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster)
[google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | --- diff --git a/anthos-onprem-terraform/abm_user_cluster_manuallb/README.md b/anthos-onprem-terraform/abm_user_cluster_manuallb/README.md new file mode 100644 index 00000000..cba6b4fe --- /dev/null +++ b/anthos-onprem-terraform/abm_user_cluster_manuallb/README.md @@ -0,0 +1,93 @@ +## Create Anthos on bare metal **user** clusters (ManualLB) with Terraform + +This sample is an example of how to create an Anthos on bare metal +**user cluster** in **ManualLB** mode using the +**`google_gkeonprem_bare_metal_*`** resources of the official Google terraform +provider. This is an alternate approach _(i.e. Terraform client)_ to what is +already explained in the +[Create a user cluster using Anthos On-Prem API clients](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/creating-clusters/create-user-cluster-api#manual) +public documentation. + +The sample here assumes that you already have an **admin cluster** that will be +managing the new cluster. It also assumes that you have your own bare metal +infrastructure along with the _Manual load balancer_ setup to provision the new +cluster using this example. + +We don't provide a complete installation guide for this sample, since the Manual +load balancer setup is a pre-requisite and is dependant on the network setup of +your bare metal infrastructure. Thus, we list out general guidelines for how +to use this sample. + +> **Note:** Googlers can use [go/abm-tf-manuallb-guide](http://go/abm-tf-manuallb-guide) +> to get this sample up and running in a GCE environment with GCLB used as the +> ManualLB. This is only to enable testing this sample for demo purposes. + +--- +### Prepare + +- Decide on which admin cluster will be used to manage the new user cluster. If + you don't have one, then create a new admin cluster. +- Ensure you have the correct network setup for [**Manual load balancing** mode](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/manual-lb). +- Ensure the workstation you will be using to run terraform has access to all + the nodes of the new cluster. +--- + +### Run Terraform + +The steps that follow assumes that you already have this repo cloned locally and +have changed directory to where this samples is: +`/anthos-onprem-terraform/abm_user_cluster_manuallb`. + +- Make a copy of the `terraform.tfvars.sample` file: + + ```sh + cp terraform.tfvars.sample terraform.tfvars + ``` + +- Fill in the `terraform.tfvars` file with values appropriate to your + environment: + - **`project_id`**: The GCP project of the admin cluster and where the user + cluster will be created. + + - **`region`**: The Google Cloud region in which the Anthos On-Prem API + runs. + - **`admin_cluster_name`**: The name of the admin cluster that will manage + the new user cluster. + - **`cluster_name`**: The name to given to the new user cluster that will be + created. + - **`bare_metal_version`**: The Anthos clusters on bare metal version for + your user cluster. This must be same as the admin cluster version or one + minor version less, at most. It cannot be higher in any case - minor or + patch. + - **`control_plane_ips`**: IP addresses of the nodes that will be part of + the control plane of the cluster. + - **`worker_node_ips`**: IP addresses of the nodes that will be part of + the worker node pools of the cluster. + - **`control_plane_vip`**: The virtual IP address (VIP) that you have chosen + to configure on the load balancer for the Kubernetes API server of the + user cluster. + - **`ingress_vip`**: The virtual IP address that you have chosen to + configure on the load balancer for the ingress proxy. + - **`admin_user_emails`**: List of GCP accounts that must be given + administrator rights on the user cluster. + +- Execute terraform: + + ```sh + terraform init + ``` + ```sh + terraform plan + ``` + ```sh + terraform apply + ``` + + Once completed you will see an output as follows: + ```sh + ... + + ``` + + You can view your user cluster in the + [Anthos clusters page](https://console.cloud.google.com/anthos/clusters). diff --git a/anthos-onprem-terraform/abm_user_cluster_manuallb/main.tf b/anthos-onprem-terraform/abm_user_cluster_manuallb/main.tf new file mode 100644 index 00000000..5a20398f --- /dev/null +++ b/anthos-onprem-terraform/abm_user_cluster_manuallb/main.tf @@ -0,0 +1,157 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#[START anthos_onprem_terraform_bare_metal_user_cluster_manuallb_main] +module "enable_google_apis_primary" { + source = "terraform-google-modules/project-factory/google//modules/project_services" + version = "~> 14.0" + project_id = var.project_id + activate_apis = [ + "cloudresourcemanager.googleapis.com", + "anthos.googleapis.com", + "anthosgke.googleapis.com", + "container.googleapis.com", + "gkeconnect.googleapis.com", + "gkehub.googleapis.com", + "serviceusage.googleapis.com", + "stackdriver.googleapis.com", + "monitoring.googleapis.com", + "logging.googleapis.com", + "iam.googleapis.com", + "compute.googleapis.com", + "anthosaudit.googleapis.com", + "opsconfigmonitoring.googleapis.com", + "file.googleapis.com", + "connectgateway.googleapis.com" + ] + disable_services_on_destroy = false +} + +# Enable GKE OnPrem API +resource "google_project_service" "default" { + project = var.project_id + service = "gkeonprem.googleapis.com" + disable_on_destroy = false +} + +# Create an anthos baremetal user cluster and enroll it with the gkeonprem API +resource "google_gkeonprem_bare_metal_cluster" "default" { + provider = google-beta + name = var.cluster_name + description = "Anthos bare metal user cluster with ManualLB" + depends_on = [google_project_service.default] + location = var.region + bare_metal_version = var.bare_metal_version + admin_cluster_membership = "projects/${var.project_id}/locations/global/memberships/${var.admin_cluster_name}" + network_config { + island_mode_cidr { + service_address_cidr_blocks = ["172.26.0.0/16"] + pod_address_cidr_blocks = ["10.240.0.0/13"] + } + } + control_plane { + control_plane_node_pool_config { + node_pool_config { + operating_system = "LINUX" + dynamic "node_configs" { + for_each = var.control_plane_ips + content { + node_ip = node_configs.value + } + } + } + } + } + load_balancer { + port_config { + control_plane_load_balancer_port = 443 + } + vip_config { + control_plane_vip = var.control_plane_vip + ingress_vip = var.ingress_vip + } + manual_lb_config { + enabled = true + } + } + storage { + lvp_share_config { + lvp_config { + path = "/mnt/localpv-share" + storage_class = "local-shared" + } + shared_path_pv_count = 5 + } + lvp_node_mounts_config { + path = "/mnt/localpv-disk" + storage_class = "local-disks" + } + } + + dynamic "security_config" { + for_each = length(var.admin_user_emails) == 0 ? [] : [1] + content { + authorization { + dynamic "admin_users" { + for_each = var.admin_user_emails + content { + username = admin_users.value + } + } + } + } + } + + lifecycle { + ignore_changes = [ + annotations["onprem.cluster.gke.io/user-cluster-resource-link"], + annotations["alpha.baremetal.cluster.gke.io/cluster-metrics-webhook"], + annotations["baremetal.cluster.gke.io/operation"], + annotations["baremetal.cluster.gke.io/operation-id"], + annotations["baremetal.cluster.gke.io/start-time"], + annotations["baremetal.cluster.gke.io/upgrade-from-version"] + ] + } +} + +# Create a node pool of worker nodes for the anthos baremetal user cluster +resource "google_gkeonprem_bare_metal_node_pool" "default" { + provider = google-beta + name = "${var.cluster_name}-nodepool" + display_name = "Nodepool for ${var.cluster_name}" + bare_metal_cluster = google_gkeonprem_bare_metal_cluster.default.name + location = var.region + node_pool_config { + operating_system = "LINUX" + labels = {} + + dynamic "node_configs" { + for_each = var.worker_node_ips + content { + labels = {} + node_ip = node_configs.value + } + } + } + + lifecycle { + ignore_changes = [ + annotations["baremetal.cluster.gke.io/gke-version"], + annotations["baremetal.cluster.gke.io/version"], + ] + } +} +#[END anthos_onprem_terraform_bare_metal_user_cluster_manuallb_main] diff --git a/anthos-onprem-terraform/abm_user_cluster_manuallb/terraform.tfvars.sample b/anthos-onprem-terraform/abm_user_cluster_manuallb/terraform.tfvars.sample new file mode 100644 index 00000000..75f32ebc --- /dev/null +++ b/anthos-onprem-terraform/abm_user_cluster_manuallb/terraform.tfvars.sample @@ -0,0 +1,14 @@ +#[START anthos_onprem_terraform_bare_metal_user_cluster_manuallb_tfvars] + +project_id = "PROJECT_ID" +region = "ON_PREM_API_REGION" +admin_cluster_name = "ADMIN_CLUSTER_NAME" +cluster_name = "YOUR_USER_CLUSTER_NAME" +bare_metal_version = "VERSION" +control_plane_ips = ["CONTROLPLANE_IP_1", "CONTROLPLANE_IP_2", "CONTROLPLANE_IP_3"] +worker_node_ips = ["WORKER_IP_1", "WORKER_IP_2"] +control_plane_vip = "CONTROLPLANE_VIRTUAL_IP" +ingress_vip = "INGRESS_VIRTUAL_IP" +admin_user_emails = ["YOUR_EMAIL_ADDRESS", "ADMIN_2_EMAIL_ADDRESS"] + +#[END anthos_onprem_terraform_bare_metal_user_cluster_manuallb_tfvars] diff --git a/anthos-onprem-terraform/abm_user_cluster_manuallb/variables.tf b/anthos-onprem-terraform/abm_user_cluster_manuallb/variables.tf new file mode 100644 index 00000000..eff6158d --- /dev/null +++ b/anthos-onprem-terraform/abm_user_cluster_manuallb/variables.tf @@ -0,0 +1,125 @@ +/** + * Copyright 2023 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "project_id" { + description = "Unique identifer of the Google Cloud Project that is to be used" + type = string +} + +variable "region" { + description = <