Skip to content

Latest commit

 

History

History
181 lines (142 loc) · 11.4 KB

README.md

File metadata and controls

181 lines (142 loc) · 11.4 KB

KKP Terraform REST Provider

The following provider manage a KKP user cluster based on from the KKP Rest API.

The provider json templates under For more Information,

To access KKP Rest API, an API token of type Editor will get needed for this example, see Using Service Accounts

Architecture

Using the given example inside of any GitOps Tooling, the following workflow is given:

KKP REST-API Terraform Provider Architecture Overview

Image Source: local kkp-rest-API-Terraform-Cluster-CRD-Architecture-Drawing.drawio.xml or Google Drive

Image Source: local kkp-rest-API-Terraform-Cluster-CRD-Architecture-Drawing.drawio.xml or Google Drive

  1. Use Authentication Token provided by the KKP Service Accounts as Headers of the curl request.
  2. Talk to the KKP Rest API with the given payload, what have been rendered by the terraform module
  3. Kubermatic API transfers the API JSON payload to Cluster object and applies it against the matching Seed Cluster Kubernetes API endpoint.
  4. Seed Controller Managers use the ClusterSpec and create the necessary specs for the Control Plan creation of a KP user cluster
  5. Containerized Control Plane objects spins up (Deployments & StatefulSets) and seed controller manager creates necessary external cloud provider resources (e.g., a security group at the external cloud).

Example

  1. create a terraform.tfvars, based on in example terraform.kubevirt.example.tfvars
cd custom-components/terraform-kkp-cluster-provider
cp terraform.kubevirt.example.tfvars terraform.tfvars

# adjust variables, and add API Token 
vim terraform.tfvars

# check
cat terraform.tfvars
api_token =  "YOUR-ACCOUNT-TOKEN" #KKP Service Account Token
project_id   = "YOUR-PROJECT_ID" #KKP project ID
cluster_name = "terraform-example-dev"
  • NOTE: If more variables should be edited check variables.tf definition!
  1. Create Cluster via terraform
terraform init
terraform apply
#...

Outputs:

k8s_id = "b2sj5trqb9"
k8s_kubeconfig = "kubeconfig-b2sj5trqb9"
k8s_name = "terraform-example-dev"
k8s_url = "https://b2sj5trqb9.run-2.lab.kubermatic.io:6443"
  • NOTE: If some Error like Error: unexpected response code '503': {"error":{"code":503,"message":"Cluster components are not ready yet"}} occurs, it could be that some components needs a little more time to get finished. Just check the KKP Dashbaoard and try again terraform apply after the control plane is up.
  1. Check Cluster access via kubectl and the created kubeconfig-cluster-id-xxx:
export KUBECONFIG=kubeconfig-b2sj5trqb9
kubectl get md,ms,ma,node -A
NAMESPACE     NAME                                                              REPLICAS   AVAILABLE-REPLICAS   PROVIDER   OS       KUBELET   AGE
kube-system   machinedeployment.cluster.k8s.io/terraform-example-dev-node   3                               kubevirt   ubuntu   1.24.9    7m9s

NAMESPACE     NAME                                                                  REPLICAS   AVAILABLE-REPLICAS   PROVIDER   OS       MACHINEDEPLOYMENT                KUBELET   AGE
kube-system   machineset.cluster.k8s.io/terraform-example-dev-node-78c49b98db   3                               kubevirt   ubuntu   terraform-example-dev-node   1.24.9    7m9s

NAMESPACE     NAME                                                                     PROVIDER   OS       NODE                                              KUBELET   ADDRESS        AGE
kube-system   machine.cluster.k8s.io/terraform-example-dev-node-78c49b98db-52975   kubevirt   ubuntu   terraform-example-dev-node-78c49b98db-52975   1.24.9    10.244.3.103   7m9s
kube-system   machine.cluster.k8s.io/terraform-example-dev-node-78c49b98db-7vsmj   kubevirt   ubuntu   terraform-example-dev-node-78c49b98db-7vsmj   1.24.9    10.244.5.42    7m9s
kube-system   machine.cluster.k8s.io/terraform-example-dev-node-78c49b98db-9cs64   kubevirt   ubuntu   terraform-example-dev-node-78c49b98db-9cs64   1.24.9    10.244.2.78    7m9s

NAMESPACE   NAME                                                   STATUS     ROLES    AGE   VERSION
            node/terraform-example-dev-node-78c49b98db-52975   NotReady   <none>   17s   v1.24.9
            node/terraform-example-dev-node-78c49b98db-7vsmj   NotReady   <none>   7s    v1.24.9
            node/terraform-example-dev-node-78c49b98db-9cs64   NotReady   <none>   17s   v1.24.9
  • NOTE: If some NODE objects are not shown up, it could take up 10 min to provision the whole nodes, depending on the datacenter utilization. You could also try watch kubectl get md,ms,ma,node -A
  1. Destroy a Cluster
terraform destroy
Plan: 0 to add, 0 to change, 6 to destroy.

Changes to Outputs:
  - k8s_id         = "b2sj5trqb9" -> null
  - k8s_kubeconfig = "kubeconfig-b2sj5trqb9" -> null
  - k8s_name       = "terraform-example-dev" -> null
  - k8s_url        = "https://b2sj5trqb9.run-2.lab.kubermatic.io:6443" -> null

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

null_resource.kubeconfig: Destroying... [id=3357773382106992637]
time_sleep.wait_20_seconds: Destroying... [id=2023-02-17T23:21:35Z]
time_sleep.wait_20_seconds: Destruction complete after 0s
null_resource.kubeconfig: Destruction complete after 0s
restapi_object.machinedeployment_create: Destroying... [id=terraform-example-dev-node]
restapi_object.machinedeployment_create: Destruction complete after 0s
time_sleep.wait_60_seconds: Destroying... [id=2023-02-17T23:22:15Z]
restapi_object.cluster_create: Destroying... [id=b2sj5trqb9]
time_sleep.wait_60_seconds: Destruction complete after 0s
null_resource.previous: Destroying... [id=2989688693603946775]
null_resource.previous: Destruction complete after 0s
restapi_object.cluster_create: Destruction complete after 1s

Destroy complete! Resources: 6 destroyed.

Terraform Provider Docs

Generated via: terraform-docs markdown table --output-file README.md --output-mode inject ./

Requirements

Name Version
restapi 1.18.2

Providers

Name Version
local 2.4.0
null 3.2.1
restapi 1.18.2

Modules

No modules.

Resources

Name Type
null_resource.kubeconfig resource
null_resource.wait_cluster_creation resource
restapi_object.addon_create resource
restapi_object.cluster_apply resource
restapi_object.machinedeployment_create resource
local_file.common_script data source

Inputs

Name Description Type Default Required
api_base_url KKP base URL string "https://mgmt-prod.cp.3ascloud.de/api" no
api_token KKP admin token, see https://docs.kubermatic.com/kubermatic/v2.21/architecture/concept/kkp-concepts/service-account/using-service-account/ string n/a yes
cluster_name User Cluster Name string "test-tf-cluster" no
cluster_spec_folder Folder what contains the cluster spec jsons string "kubevirt_cluster_example" no
cni_plugin CNI Pluging name: cilium or canal string "cilium" no
cni_version Version of the CNI Plugin string "1.13.8" no
common_bash_file_path Helper file for some bash functions string "_common.sh" no
credential_preset_name Preset to use while creating the user cluster string "TODO-ADD-PRESET-NAME" no
dc Datacenter name at a Kubermatic Seed string "fra-prod" no
kubernetes_version User Cluster Kubernetes Version string "1.26.9" no
kubevirt_machine_spec A selection of parameter for a kubevirt machine object specification. 
object({
    cpus         = number
    memory       = string
    disk_size    = string
    os_image_url = string
    primary_disk_storage_class = string
  })
 
{
  "cpus": 8,
  "disk_size": "150Gi",
  "memory": "32768Mi",
  "os_image_url": "http://TODO.your.image.url/vms/ubuntu-22.04.img",
  "primary_disk_storage_class": "TODO_YOUR_STORAGE_CLASS"
}
 no
machine_osp_name Name of the selected Operating System Profile (OSP) for the machine controller string "osp-ubuntu" no
machine_replica Replica of initial created Machines number 2 no
project_id KKP Project Id string "TODO_PROJECT_ID" no

Outputs

Name Description
k8s_id K8S ID
k8s_kubeconfig K8S kubeconfig
k8s_name K8S Name
k8s_url K8S URL