feature: add user cluster create with terraform sample (manuallb)

anthos-onprem-terraform/README.md

@@ -11,9 +11,10 @@ For more information, see the reference documentation for each resource.
 
 #### Anthos clusters on bare metal (ABM)
 
-| Type             | Sample _(by loadbalancer type)_                | Terraform resources | 
-| ---------------- | ---------------------------------------------- | ------------------- |
-| **user** cluster | Bundled [MetalLB](./abm_user_cluster_metallb/) | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster) </br> [google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | 
+| Type             | Sample _(by loadbalancer type)_                  | Terraform resources | 
+| ---------------- | ----------------------------------------------   | ------------------- |
+| **user** cluster | Bundled [MetalLB](./abm_user_cluster_metallb/)   | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster) </br> [google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | 
+| **user** cluster | [ManualLB](./abm_user_cluster_manuallb/)         | [google_gkeonprem_bare_metal_cluster](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_cluster) </br> [google_gkeonprem_bare_metal_node_pool](https://registry.terraform.io/providers/hashicorp/google-beta/latest/docs/resources/gkeonprem_bare_metal_node_pool) | 
 
 ---
 

anthos-onprem-terraform/abm_user_cluster_manuallb/README.md

@@ -0,0 +1,93 @@
+## Create Anthos on bare metal **user** clusters (ManualLB) with Terraform
+
+This sample is an example of how to create an Anthos on bare metal
+**user cluster** in **ManualLB** mode using the
+**`google_gkeonprem_bare_metal_*`** resources of the official Google terraform
+provider. This is an alternate approach _(i.e. Terraform client)_ to what is
+already explained in the
+[Create a user cluster using Anthos On-Prem API clients](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/creating-clusters/create-user-cluster-api#manual)
+public documentation.
+
+The sample here assumes that you already have an **admin cluster** that will be
+managing the new cluster. It also assumes that you have your own bare metal
+infrastructure along with the _Manual load balancer_ setup to provision the new
+cluster using this example.
+
+We don't provide a complete installation guide for this sample, since the Manual
+load balancer setup is a pre-requisite and is dependant on the network setup of
+your bare metal infrastructure. Thus, we list out general guidelines for how
+to use this sample.
+
+> **Note:** Googlers can use [go/abm-tf-manuallb-guide](http://go/abm-tf-manuallb-guide)
+> to get this sample up and running in a GCE environment with GCLB used as the
+> ManualLB. This is only to enable testing this sample for demo purposes.
+
+---
+### Prepare
+
+- Decide on which admin cluster will be used to manage the new user cluster. If
+  you don't have one, then create a new admin cluster.
+- Ensure you have the correct network setup for [**Manual load balancing** mode](https://cloud.google.com/anthos/clusters/docs/bare-metal/latest/installing/manual-lb).
+- Ensure the workstation you will be using to run terraform has access to all
+  the nodes of the new cluster.
+---
+
+### Run Terraform
+
+The steps that follow assumes that you already have this repo cloned locally and
+have changed directory to where this samples is:
+`<REPO_ROOT_DIR>/anthos-onprem-terraform/abm_user_cluster_manuallb`.
+
+- Make a copy of the `terraform.tfvars.sample` file:
+
+    ```sh
+    cp terraform.tfvars.sample terraform.tfvars
+    ```
+
+- Fill in the `terraform.tfvars` file with values appropriate to your
+  environment:
+  - **`project_id`**: The GCP project of the admin cluster and where the user
+    cluster will be created.
+
+  - **`region`**: The Google Cloud region in which the Anthos On-Prem API
+    runs.
+  - **`admin_cluster_name`**: The name of the admin cluster that will manage
+    the new user cluster.
+  - **`cluster_name`**: The name to given to the new user cluster that will be
+    created.
+  - **`bare_metal_version`**: The Anthos clusters on bare metal version for
+    your user cluster. This must be same as the admin cluster version or one
+    minor version less, at most. It cannot be higher in any case - minor or
+    patch.
+  - **`control_plane_ips`**: IP addresses of the nodes that will be part of
+    the control plane of the cluster.
+  - **`worker_node_ips`**: IP addresses of the nodes that will be part of
+    the worker node pools of the cluster.
+  - **`control_plane_vip`**: The virtual IP address (VIP) that you have chosen
+    to configure on the load balancer for the Kubernetes API server of the
+    user cluster.
+  - **`ingress_vip`**: The virtual IP address that you have chosen to
+    configure on the load balancer for the ingress proxy.
+  - **`admin_user_emails`**: List of GCP accounts that must be given
+    administrator rights on the user cluster.
+
+- Execute terraform:
+
+    ```sh
+    terraform init
+    ```
+    ```sh
+    terraform plan
+    ```
+    ```sh
+    terraform apply
+    ```
+
+    Once completed you will see an output as follows:
+    ```sh
+    ...
+
+    ```
+
+    You can view your user cluster in the
+    [Anthos clusters page](https://console.cloud.google.com/anthos/clusters).

anthos-onprem-terraform/abm_user_cluster_manuallb/main.tf

@@ -0,0 +1,157 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#[START anthos_onprem_terraform_bare_metal_user_cluster_manuallb_main]
+module "enable_google_apis_primary" {
+  source     = "terraform-google-modules/project-factory/google//modules/project_services"
+  version    = "~> 14.0"
+  project_id = var.project_id
+  activate_apis = [
+    "cloudresourcemanager.googleapis.com",
+    "anthos.googleapis.com",
+    "anthosgke.googleapis.com",
+    "container.googleapis.com",
+    "gkeconnect.googleapis.com",
+    "gkehub.googleapis.com",
+    "serviceusage.googleapis.com",
+    "stackdriver.googleapis.com",
+    "monitoring.googleapis.com",
+    "logging.googleapis.com",
+    "iam.googleapis.com",
+    "compute.googleapis.com",
+    "anthosaudit.googleapis.com",
+    "opsconfigmonitoring.googleapis.com",
+    "file.googleapis.com",
+    "connectgateway.googleapis.com"
+  ]
+  disable_services_on_destroy = false
+}
+
+# Enable GKE OnPrem API
+resource "google_project_service" "default" {
+  project            = var.project_id
+  service            = "gkeonprem.googleapis.com"
+  disable_on_destroy = false
+}
+
+# Create an anthos baremetal user cluster and enroll it with the gkeonprem API
+resource "google_gkeonprem_bare_metal_cluster" "default" {
+  provider                 = google-beta
+  name                     = var.cluster_name
+  description              = "Anthos bare metal user cluster with ManualLB"
+  depends_on               = [google_project_service.default]
+  location                 = var.region
+  bare_metal_version       = var.bare_metal_version
+  admin_cluster_membership = "projects/${var.project_id}/locations/global/memberships/${var.admin_cluster_name}"
+  network_config {
+    island_mode_cidr {
+      service_address_cidr_blocks = ["172.26.0.0/16"]
+      pod_address_cidr_blocks     = ["10.240.0.0/13"]
+    }
+  }
+  control_plane {
+    control_plane_node_pool_config {
+      node_pool_config {
+        operating_system = "LINUX"
+        dynamic "node_configs" {
+          for_each = var.control_plane_ips
+          content {
+            node_ip = node_configs.value
+          }
+        }
+      }
+    }
+  }
+  load_balancer {
+    port_config {
+      control_plane_load_balancer_port = 443
+    }
+    vip_config {
+      control_plane_vip = var.control_plane_vip
+      ingress_vip       = var.ingress_vip
+    }
+    manual_lb_config {
+      enabled = true
+    }
+  }
+  storage {
+    lvp_share_config {
+      lvp_config {
+        path          = "/mnt/localpv-share"
+        storage_class = "local-shared"
+      }
+      shared_path_pv_count = 5
+    }
+    lvp_node_mounts_config {
+      path          = "/mnt/localpv-disk"
+      storage_class = "local-disks"
+    }
+  }
+
+  dynamic "security_config" {
+    for_each = length(var.admin_user_emails) == 0 ? [] : [1]
+    content {
+      authorization {
+        dynamic "admin_users" {
+          for_each = var.admin_user_emails
+          content {
+            username = admin_users.value
+          }
+        }
+      }
+    }
+  }
+
+  lifecycle {
+    ignore_changes = [
+      annotations["onprem.cluster.gke.io/user-cluster-resource-link"],
+      annotations["alpha.baremetal.cluster.gke.io/cluster-metrics-webhook"],
+      annotations["baremetal.cluster.gke.io/operation"],
+      annotations["baremetal.cluster.gke.io/operation-id"],
+      annotations["baremetal.cluster.gke.io/start-time"],
+      annotations["baremetal.cluster.gke.io/upgrade-from-version"]
+    ]
+  }
+}
+
+# Create a node pool of worker nodes for the anthos baremetal user cluster
+resource "google_gkeonprem_bare_metal_node_pool" "default" {
+  provider           = google-beta
+  name               = "${var.cluster_name}-nodepool"
+  display_name       = "Nodepool for ${var.cluster_name}"
+  bare_metal_cluster = google_gkeonprem_bare_metal_cluster.default.name
+  location           = var.region
+  node_pool_config {
+    operating_system = "LINUX"
+    labels           = {}
+
+    dynamic "node_configs" {
+      for_each = var.worker_node_ips
+      content {
+        labels  = {}
+        node_ip = node_configs.value
+      }
+    }
+  }
+
+  lifecycle {
+    ignore_changes = [
+      annotations["baremetal.cluster.gke.io/gke-version"],
+      annotations["baremetal.cluster.gke.io/version"],
+    ]
+  }
+}
+#[END anthos_onprem_terraform_bare_metal_user_cluster_manuallb_main]

anthos-onprem-terraform/abm_user_cluster_manuallb/terraform.tfvars.sample

@@ -0,0 +1,14 @@
+#[START anthos_onprem_terraform_bare_metal_user_cluster_manuallb_tfvars]
+
+project_id          = "PROJECT_ID"
+region              = "ON_PREM_API_REGION"
+admin_cluster_name  = "ADMIN_CLUSTER_NAME"
+cluster_name        = "YOUR_USER_CLUSTER_NAME"
+bare_metal_version  = "VERSION"
+control_plane_ips   = ["CONTROLPLANE_IP_1", "CONTROLPLANE_IP_2", "CONTROLPLANE_IP_3"]
+worker_node_ips     = ["WORKER_IP_1", "WORKER_IP_2"]
+control_plane_vip   = "CONTROLPLANE_VIRTUAL_IP"
+ingress_vip         = "INGRESS_VIRTUAL_IP"
+admin_user_emails   = ["YOUR_EMAIL_ADDRESS", "ADMIN_2_EMAIL_ADDRESS"]
+
+#[END anthos_onprem_terraform_bare_metal_user_cluster_manuallb_tfvars]