Skip to content

Commit

Permalink
update equinox terraform code to fix kubespray CI (kubernetes-sigs#9702)
Browse files Browse the repository at this point in the history
* add terraform lock files to ignore list

* move contrib/terraform/metal to contrib/terraform/equinix to reflect upstream change
  • Loading branch information
cristicalin authored Jan 27, 2023
1 parent 6881398 commit 64dbf2e
Show file tree
Hide file tree
Showing 13 changed files with 85 additions and 66 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ contrib/offline/offline-files.tar.gz
*.bak
*.tfstate
*.tfstate.backup
*.lock.hcl
.terraform/
contrib/terraform/aws/credentials.tfvars
.terraform.lock.hcl
Expand Down
4 changes: 2 additions & 2 deletions .gitlab-ci/terraform.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,11 @@ tf-validate-openstack:
PROVIDER: openstack
CLUSTER: $CI_COMMIT_REF_NAME

tf-validate-metal:
tf-validate-equinix:
extends: .terraform_validate
variables:
TF_VERSION: $TERRAFORM_VERSION
PROVIDER: metal
PROVIDER: equinix
CLUSTER: $CI_COMMIT_REF_NAME

tf-validate-aws:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This will install a Kubernetes cluster on Equinix Metal. It should work in all l
The terraform configuration inspects variables found in
[variables.tf](variables.tf) to create resources in your Equinix Metal project.
There is a [python script](../terraform.py) that reads the generated`.tfstate`
file to generate a dynamic inventory that is consumed by [cluster.yml](../../..//cluster.yml)
file to generate a dynamic inventory that is consumed by [cluster.yml](../../../cluster.yml)
to actually install Kubernetes with Kubespray.

### Kubernetes Nodes
Expand Down Expand Up @@ -60,16 +60,16 @@ Terraform will be used to provision all of the Equinix Metal resources with base
Create an inventory directory for your cluster by copying the existing sample and linking the `hosts` script (used to build the inventory based on Terraform state):

```ShellSession
cp -LRp contrib/terraform/metal/sample-inventory inventory/$CLUSTER
cp -LRp contrib/terraform/equinix/sample-inventory inventory/$CLUSTER
cd inventory/$CLUSTER
ln -s ../../contrib/terraform/metal/hosts
ln -s ../../contrib/terraform/equinix/hosts
```

This will be the base for subsequent Terraform commands.

#### Equinix Metal API access

Your Equinix Metal API key must be available in the `PACKET_AUTH_TOKEN` environment variable.
Your Equinix Metal API key must be available in the `METAL_AUTH_TOKEN` environment variable.
This key is typically stored outside of the code repo since it is considered secret.
If someone gets this key, they can startup/shutdown hosts in your project!

Expand All @@ -80,10 +80,12 @@ The Equinix Metal Project ID associated with the key will be set later in `clust

For more information about the API, please see [Equinix Metal API](https://metal.equinix.com/developers/api/).

For more information about terraform provider authentication, please see [the equinix provider documentation](https://registry.terraform.io/providers/equinix/equinix/latest/docs).

Example:

```ShellSession
export PACKET_AUTH_TOKEN="Example-API-Token"
export METAL_AUTH_TOKEN="Example-API-Token"
```

Note that to deploy several clusters within the same project you need to use [terraform workspace](https://www.terraform.io/docs/state/workspaces.html#using-workspaces).
Expand All @@ -101,7 +103,7 @@ This helps when identifying which hosts are associated with each cluster.
While the defaults in variables.tf will successfully deploy a cluster, it is recommended to set the following values:

- cluster_name = the name of the inventory directory created above as $CLUSTER
- metal_project_id = the Equinix Metal Project ID associated with the Equinix Metal API token above
- equinix_metal_project_id = the Equinix Metal Project ID associated with the Equinix Metal API token above

#### Enable localhost access

Expand All @@ -119,12 +121,13 @@ Once the Kubespray playbooks are run, a Kubernetes configuration file will be wr

In the cluster's inventory folder, the following files might be created (either by Terraform
or manually), to prevent you from pushing them accidentally they are in a
`.gitignore` file in the `terraform/metal` directory :
`.gitignore` file in the `contrib/terraform/equinix` directory :

- `.terraform`
- `.tfvars`
- `.tfstate`
- `.tfstate.backup`
- `.lock.hcl`

You can still add them manually if you want to.

Expand All @@ -135,7 +138,7 @@ plugins. This is accomplished as follows:

```ShellSession
cd inventory/$CLUSTER
terraform init ../../contrib/terraform/metal
terraform -chdir=../../contrib/terraform/metal init -var-file=cluster.tfvars
```

This should finish fairly quickly telling you Terraform has successfully initialized and loaded necessary modules.
Expand All @@ -146,7 +149,7 @@ You can apply the Terraform configuration to your cluster with the following com
issued from your cluster's inventory directory (`inventory/$CLUSTER`):

```ShellSession
terraform apply -var-file=cluster.tfvars ../../contrib/terraform/metal
terraform -chdir=../../contrib/terraform/equinix apply -var-file=cluster.tfvars
export ANSIBLE_HOST_KEY_CHECKING=False
ansible-playbook -i hosts ../../cluster.yml
```
Expand All @@ -156,7 +159,7 @@ ansible-playbook -i hosts ../../cluster.yml
You can destroy your new cluster with the following command issued from the cluster's inventory directory:

```ShellSession
terraform destroy -var-file=cluster.tfvars ../../contrib/terraform/metal
terraform -chdir=../../contrib/terraform/equinix destroy -var-file=cluster.tfvars
```

If you've started the Ansible run, it may also be a good idea to do some manual cleanup:
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -1,62 +1,57 @@
# Configure the Equinix Metal Provider
provider "metal" {
}

resource "metal_ssh_key" "k8s" {
resource "equinix_metal_ssh_key" "k8s" {
count = var.public_key_path != "" ? 1 : 0
name = "kubernetes-${var.cluster_name}"
public_key = chomp(file(var.public_key_path))
}

resource "metal_device" "k8s_master" {
depends_on = [metal_ssh_key.k8s]
resource "equinix_metal_device" "k8s_master" {
depends_on = [equinix_metal_ssh_key.k8s]

count = var.number_of_k8s_masters
hostname = "${var.cluster_name}-k8s-master-${count.index + 1}"
plan = var.plan_k8s_masters
facilities = [var.facility]
operating_system = var.operating_system
billing_cycle = var.billing_cycle
project_id = var.metal_project_id
project_id = var.equinix_metal_project_id
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_control_plane", "etcd", "kube_node"]
}

resource "metal_device" "k8s_master_no_etcd" {
depends_on = [metal_ssh_key.k8s]
resource "equinix_metal_device" "k8s_master_no_etcd" {
depends_on = [equinix_metal_ssh_key.k8s]

count = var.number_of_k8s_masters_no_etcd
hostname = "${var.cluster_name}-k8s-master-${count.index + 1}"
plan = var.plan_k8s_masters_no_etcd
facilities = [var.facility]
operating_system = var.operating_system
billing_cycle = var.billing_cycle
project_id = var.metal_project_id
project_id = var.equinix_metal_project_id
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_control_plane"]
}

resource "metal_device" "k8s_etcd" {
depends_on = [metal_ssh_key.k8s]
resource "equinix_metal_device" "k8s_etcd" {
depends_on = [equinix_metal_ssh_key.k8s]

count = var.number_of_etcd
hostname = "${var.cluster_name}-etcd-${count.index + 1}"
plan = var.plan_etcd
facilities = [var.facility]
operating_system = var.operating_system
billing_cycle = var.billing_cycle
project_id = var.metal_project_id
project_id = var.equinix_metal_project_id
tags = ["cluster-${var.cluster_name}", "etcd"]
}

resource "metal_device" "k8s_node" {
depends_on = [metal_ssh_key.k8s]
resource "equinix_metal_device" "k8s_node" {
depends_on = [equinix_metal_ssh_key.k8s]

count = var.number_of_k8s_nodes
hostname = "${var.cluster_name}-k8s-node-${count.index + 1}"
plan = var.plan_k8s_nodes
facilities = [var.facility]
operating_system = var.operating_system
billing_cycle = var.billing_cycle
project_id = var.metal_project_id
project_id = var.equinix_metal_project_id
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_node"]
}

16 changes: 16 additions & 0 deletions contrib/terraform/equinix/output.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
output "k8s_masters" {
value = equinix_metal_device.k8s_master.*.access_public_ipv4
}

output "k8s_masters_no_etc" {
value = equinix_metal_device.k8s_master_no_etcd.*.access_public_ipv4
}

output "k8s_etcds" {
value = equinix_metal_device.k8s_etcd.*.access_public_ipv4
}

output "k8s_nodes" {
value = equinix_metal_device.k8s_node.*.access_public_ipv4
}

13 changes: 13 additions & 0 deletions contrib/terraform/equinix/provider.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
terraform {
required_version = ">= 1.0.0"
required_providers {
equinix = {
source = "equinix/equinix"
version = ">=1.11.0"
}
}
}

# Configure the Equinix Metal Provider
provider "equinix" {
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
cluster_name = "mycluster"

# Your Equinix Metal project ID. See hhttps://metal.equinix.com/developers/docs/accounts/
metal_project_id = "Example-API-Token"
equinix_metal_project_id = "Example-Project-Id"

# The public SSH key to be uploaded into authorized_keys in bare metal Equinix Metal nodes provisioned
# leave this value blank if the public key is already setup in the Equinix Metal project
Expand All @@ -12,6 +12,9 @@ public_key_path = "~/.ssh/id_rsa.pub"
# cluster location
facility = "ewr1"

# operating_system
operating_system = "ubuntu_22_04"

# standalone etcds
number_of_etcd = 0

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@ variable "cluster_name" {
default = "kubespray"
}

variable "metal_project_id" {
variable "equinix_metal_project_id" {
description = "Your Equinix Metal project ID. See https://metal.equinix.com/developers/docs/accounts/"
}

variable "operating_system" {
default = "ubuntu_20_04"
default = "ubuntu_22_04"
}

variable "public_key_path" {
Expand Down
16 changes: 0 additions & 16 deletions contrib/terraform/metal/output.tf

This file was deleted.

9 changes: 0 additions & 9 deletions contrib/terraform/metal/versions.tf

This file was deleted.

29 changes: 21 additions & 8 deletions contrib/terraform/terraform.py
Original file line number Diff line number Diff line change
Expand Up @@ -194,9 +194,19 @@ def parse_bool(string_form):
else:
raise ValueError('could not convert %r to a bool' % string_form)


@parses('metal_device')
def metal_device(resource, tfvars=None):
def sanitize_groups(groups):
_groups = []
chars_to_replace = ['+', '-', '=', '.', '/', ' ']
for i in groups:
_i = i
for char in chars_to_replace:
_i = _i.replace(char, '_')
_groups.append(_i)
groups.clear()
groups.extend(_groups)

@parses('equinix_metal_device')
def equinix_metal_device(resource, tfvars=None):
raw_attrs = resource['primary']['attributes']
name = raw_attrs['hostname']
groups = []
Expand All @@ -220,21 +230,22 @@ def metal_device(resource, tfvars=None):
'ipv6_address': raw_attrs['network.1.address'],
'public_ipv6': raw_attrs['network.1.address'],
'private_ipv4': raw_attrs['network.2.address'],
'provider': 'metal',
'provider': 'equinix',
}

if raw_attrs['operating_system'] == 'flatcar_stable':
# For Flatcar set the ssh_user to core
attrs.update({'ansible_ssh_user': 'core'})

# add groups based on attrs
groups.append('metal_operating_system=' + attrs['operating_system'])
groups.append('metal_locked=%s' % attrs['locked'])
groups.append('metal_state=' + attrs['state'])
groups.append('metal_plan=' + attrs['plan'])
groups.append('equinix_metal_operating_system_%s' % attrs['operating_system'])
groups.append('equinix_metal_locked_%s' % attrs['locked'])
groups.append('equinix_metal_state_%s' % attrs['state'])
groups.append('equinix_metal_plan_%s' % attrs['plan'])

# groups specific to kubespray
groups = groups + attrs['tags']
sanitize_groups(groups)

return name, attrs, groups

Expand Down Expand Up @@ -334,6 +345,8 @@ def openstack_host(resource, module_name):
for group in attrs['metadata'].get('kubespray_groups', "").split(","):
groups.append(group)

sanitize_groups(groups)

return name, attrs, groups


Expand Down

0 comments on commit 64dbf2e

Please sign in to comment.