Need help in decreasing the complexity and amount of downloads for Ceph snippet/ISO storage for cloud-init based VM's... #678

ab17182 · 2023-11-03T20:59:16Z

ab17182
Nov 3, 2023

Hi everyone,

I am hoping someone can help me out. I have a Ceph pool created for 3 node cluster which is used primarily for a snippets/ISO datastore for spinning up cloud-init based Debian 11 VM's. The issue I am running into is that I think I am overcomplicating the amount of downloads for the VM's I'm creating and am hoping someone can take a gander at what I am doing and see if there's an easier way. Note that this is working as expected but I just wanted to decrease the amount of times I am fetching and uploading an ISO and snippet file to the Ceph pool.

I'm using the latest 0.37.0 release as well (just as an FYI).

Here's my files.tf:
`resource "proxmox_virtual_environment_file" "debian_cloud_image" {
content_type = "iso"
datastore_id = "cephfs"
node_name = "proxmox1"

source_file {
path = "https://cdimage.debian.org/images/cloud/bullseye/20221219-1234/debian-11-genericcloud-amd64-20221219-1234.qcow2"
file_name = "debian-11-genericcloud-amd64-20221219-1234.img"
checksum = "ba0237232247948abf7341a495dec009702809aa7782355a1b35c112e75cee81"
}
}
resource "proxmox_virtual_environment_file" "cloud_config" {
content_type = "snippets"
datastore_id = "cephfs"
node_name = "proxmox1"

source_file {
path = "cloud-init/user-data.yml"
}
}

resource "proxmox_virtual_environment_file" "debian_cloud_image2" {
content_type = "iso"
datastore_id = "cephfs"
node_name = "proxmox2"

source_file {
path = "https://cdimage.debian.org/images/cloud/bullseye/20221219-1234/debian-11-genericcloud-amd64-20221219-1234.qcow2"
file_name = "debian-11-genericcloud-amd64-20221219-1234.img"
checksum = "ba0237232247948abf7341a495dec009702809aa7782355a1b35c112e75cee81"
}
}
resource "proxmox_virtual_environment_file" "cloud_config2" {
content_type = "snippets"
datastore_id = "cephfs"
node_name = "proxmox2"

source_file {
path = "cloud-init/user-data.yml"
}
}

resource "proxmox_virtual_environment_file" "debian_cloud_image3" {
content_type = "iso"
datastore_id = "cephfs"
node_name = "proxmox3"

source_file {
path = "https://cdimage.debian.org/images/cloud/bullseye/20221219-1234/debian-11-genericcloud-amd64-20221219-1234.qcow2"
file_name = "debian-11-genericcloud-amd64-20221219-1234.img"
checksum = "ba0237232247948abf7341a495dec009702809aa7782355a1b35c112e75cee81"
}
}
resource "proxmox_virtual_environment_file" "cloud_config3" {
content_type = "snippets"
datastore_id = "cephfs"
node_name = "proxmox3"

source_file {
path = "cloud-init/user-data.yml"
}
}`

The above will end up downloading the Debian cloud-init image 3 times instead of just getting it once.

Here's my virtual_machines.tf:
`resource "proxmox_virtual_environment_vm" "k8s_cp_01" {
name = "k8s-cp-01"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox1"
vm_id = "100"

cpu {
cores = 2
}

memory {
dedicated = "4096"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "3A:53:4E:50:13:F6"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

 ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_01" {

name = "k8s-worker-01"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox1"
vm_id = "101"

cpu {
cores = 1
}

memory {
dedicated = "4096"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "2A:B4:2E:60:BD:5F"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_02" {

name = "k8s-worker-02"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox1"
vm_id = "102"

cpu {
cores = 1
}

memory {
dedicated = "4096"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "CA:1D:F9:41:75:BA"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_03" {

name = "k8s-worker-03"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox2"
vm_id = "103"

cpu {
cores = 1
}

memory {
dedicated = "6144"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "6A:EB:AB:27:EA:53"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image2.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config2.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_04" {

name = "k8s-worker-04"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox2"
vm_id = "104"

cpu {
cores = 1
}

memory {
dedicated = "6144"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "E6:3B:B6:D1:D4:2E"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image2.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config2.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_05" {

name = "k8s-worker-05"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox3"
vm_id = "105"

cpu {
cores = 1
}

memory {
dedicated = "6144"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "22:D1:D2:D0:51:35"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image3.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config3.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}
resource "proxmox_virtual_environment_vm" "k8s_worker_06" {

name = "k8s-worker-06"
description = "Managed by Terraform"
tags = ["terraform"]
node_name = "proxmox3"
vm_id = "106"

cpu {
cores = 1
}

memory {
dedicated = "6144"
}

agent {
enabled = true
}

network_device {
bridge = "vmbr0"
mac_address = "2E:B1:38:11:0E:A6"
}

disk {
datastore_id = "local-zfs"
file_id = proxmox_virtual_environment_file.debian_cloud_image3.id
interface = "scsi0"
size = "32"
}

serial_device {} # The Debian cloud image expects a serial port to be present

operating_system {
type = "l26" # Linux Kernel 2.6 - 5.X.
}

initialization {
datastore_id = "local-zfs"
user_data_file_id = proxmox_virtual_environment_file.cloud_config3.id

ip_config {
  ipv4 {
    address = "dhcp"
  }
}

}
}`

What you'll notice is that the disk and cloud-inits are pointing to their respective node's datastores. It's because the VM's disks are stored on each node of the cluster as I don't have pooled storage for these nodes. Each node has a local ZFS datastore for VM disks.

Is there a way to eliminate the amount of downloads and limit it to one and still have each VM in the virtual_machines.tf define their respective "file_id" and "user_data_file_id" locations on their respective node? I'm looking for something like "node_name" to be available for both the disk {} and initialization {} arguments. If that was the case, I could just download once to the Ceph pool and then directly define that the 2 arguments are located in the specific node.

Hope this makes sense. I'm just goofing around with this in my homelab and am loving it. It's used to kick off an ansible to kubespray method so I can quickly create/destroy a k8s cluster for development and testing. Keep up the great work!

otopetrik · 2023-11-04T02:25:28Z

otopetrik
Nov 4, 2023

Is there a way to eliminate the amount of downloads and limit it to one and still have each VM in the virtual_machines.tf define their respective "file_id" and "user_data_file_id" locations on their respective node?

yes.

I'm looking for something like "node_name" to be available for both the disk {} and initialization {} arguments. If that was the case, I could just download once to the Ceph pool and then directly define that the 2 arguments are located in the specific node.

That would not work. node_name is specified for the whole VM, it selects the host machine, which runs QEMU/KVM process. disk and initialization have datatore_id which selects which datastore (accessible from node_name host) is used for given disk.

Ceph datastore should be accessible to from multiple machines. The correct way is to download the image only once, and for each VM use file_id, which is usable on given node_name.

According to the documentation, the ID format for proxmox_virtual_environment_file is node_name:datastore_id/content_type/file_name.

Replacing

 file_id = proxmox_virtual_environment_file.debian_cloud_image2.id

with

file_id = "proxmox2:${trimprefix(proxmox_virtual_environment_file.debian_cloud_image.id, "proxmox1:")}"

should work.
(I do not have a Ceph install, so it is not verified)
Because the expression references the original proxmox_virtual_environment_file resource, even creation of a VM on proxmox2 should wait until image is downloaded on proxmox1.

Hope this makes sense. I'm just goofing around with this in my homelab and am loving it. It's used to kick off an ansible to kubespray method so I can quickly create/destroy a k8s cluster for development and testing. Keep up the great work!

Few possibly useful hints:

With guest agent enabled, make sure to read (recently added) Qemu guest agent section of proxmox_virtual_environment_vm documentation. Debian cloud images do not have guest agent installed by default, and installing qemu-guest-agent package does not seem to start it. This worked for me.

Argument vm_id is optional, it is usually easier to let Proxmox allocate VM ids (and possibly safer, because vm_id should not collide). And Terraform has for_each documentation, tutorial which can be used to generate multiple VMs using single resource block (and one variable).

Terraform providers do not have depends_on (terraform bug), it is usually impossible to create resources from multiple layers in single terraform project. An attempt to provison VMs using Proxmox provider and provision Kubernetes resource using Kubernetes provider together in one terraform project would fail. The Kubernetes provider would not be able to connect to not yet existing Kubernetes cluster (which is needed for planning purposes (to compare wanted resources with state of the cluster).
Solution for cases "create VMs using terraform and configure something inside new VMs using terraform" is usually to use additional tooling, like terragrunt.

Proxmox does not support using disks "without VM", but it is possible to have data disks survive VM re-creation, see attached disks (experimental).

0 replies

ab17182 · 2023-11-04T03:49:32Z

ab17182
Nov 4, 2023
Author

Oh wow, that was a quick response! Will test and get back to you. This was excellent info and MUCH appreciated!

4 replies

ab17182 Nov 4, 2023
Author

Also, for qemu guest agent, I pass that as an install in my cloud-init and it seems to work a treat. This is just a snippet called user-data.yml inside a cloud-init directory:

#cloud-config
package_upgrade: true
packages: qemu-guest-agent

timezone: America/New_York
users:

default
name: adam
ssh-authorized-keys:
- ssh-rsa [ssh_public_key] adam@[some_machine]
  passwd: "[base_64_password_hash]"
  groups: [adm, cdrom, dip, plugdev, lxd, sudo]
  lock-passwd: false
  sudo: ALL=(ALL) NOPASSWD:ALL
  shell: /bin/bash

runcmd:
- sed -i -e '/^PermitRootLogin/s/^.*$/PermitRootLogin yes/' /etc/ssh/sshd_config
- sed -i -e '$aAllowUsers adam' /etc/ssh/sshd_config
- sudo killall -HUP sshd

power_state:
delay: now
mode: reboot
message: Rebooting after cloud-init completion
condition: true

otopetrik Nov 4, 2023

Just installing the package did not work for me, it was not started automatically. Had to also use:

runcmd:
 - [ systemctl, start, qemu-guest-agent ]

but

 mode: reboot

is enough if you want to reboot the VM anyway.

ab17182 Nov 4, 2023
Author

Odd, it works fine for me. Take a look at my initialization argument for the VM and compare to yours. When my VM comes online, it customizes it and pulls down the agent fine and what I then do is snag the IP to kick off an ansible.tf which runs the Kubespray playbook to get over the issue you're describing and using terragrunt. My repo has an example of the file structure and example files I use.

Btw, much thanks for your help! I am playing with your files.tf suggestions now and modifying the VM disk arguments to test out your hints. Will let you know how it goes.

ab17182 Nov 4, 2023
Author

OH wait! For qemu agent, try making it the ONLY package installed (i.e. take out your Apache package and make it only one line). I think there's an issue with more than one package defined. I remembered that I wanted to add open-iscsi in the past and both qemu-agent and open-iscsi weren't installing...

ab17182 · 2023-11-04T15:28:05Z

ab17182
Nov 4, 2023
Author

Ok, so I tried this:
file_id = "proxmox2:${trimprefix(proxmox_virtual_environment_file.debian_cloud_image.id, "proxmox1:")}"

And unfortunately, it doesn't create a hard disk for the VM. It did correctly create a cloud-init drive though (which is a step forward). The cloud-init drive is the drive which contains the initialization (user-data.yml) files needed to get the cloud VM personalized.

I'm thinking there's something I'm missing for the hard drive assignment parameters. Terraform spits this out when I try to run "apply":

╷
│ Error: storage 'proxmox2' does not exist
│
│
│ with proxmox_virtual_environment_vm.k8s_worker_03,
│ on virtual_machines.tf line 147, in resource "proxmox_virtual_environment_vm" "k8s_worker_03":
│ 147: resource "proxmox_virtual_environment_vm" "k8s_worker_03" {
│
╵
╷
│ Error: storage 'proxmox2' does not exist
│
│
│ with proxmox_virtual_environment_vm.k8s_worker_04,
│ on virtual_machines.tf line 198, in resource "proxmox_virtual_environment_vm" "k8s_worker_04":
│ 198: resource "proxmox_virtual_environment_vm" "k8s_worker_04" {
│
╵
╷
│ Error: storage 'proxmox3' does not exist
│
│
│ with proxmox_virtual_environment_vm.k8s_worker_05,
│ on virtual_machines.tf line 249, in resource "proxmox_virtual_environment_vm" "k8s_worker_05":
│ 249: resource "proxmox_virtual_environment_vm" "k8s_worker_05" {
│
╵
╷
│ Error: storage 'proxmox3' does not exist
│
│
│ with proxmox_virtual_environment_vm.k8s_worker_06,
│ on virtual_machines.tf line 300, in resource "proxmox_virtual_environment_vm" "k8s_worker_06":
│ 300: resource "proxmox_virtual_environment_vm" "k8s_worker_06" {
│
╵

0 replies

ab17182 · 2023-11-04T16:31:22Z

ab17182
Nov 4, 2023
Author

Ok, so this worked perfectly. Not sure how it is working because the files.tf points only to proxmox1 but it is working as expected:

main.tf -->

terraform {
  required_providers {
    proxmox = {
      source = "bpg/proxmox"
      version = "0.37.0"
    }
  }
}

provider "proxmox" {
    endpoint = "https://[IP_of_Proxmox]:8006/"
    insecure = "true"
}

files.tf -->

resource "proxmox_virtual_environment_file" "debian_cloud_image" {
  content_type = "iso"
  datastore_id = "cephfs"
  node_name    = "proxmox1"

  source_file {
    path      = "https://cdimage.debian.org/images/cloud/bullseye/20221219-1234/debian-11-genericcloud-amd64-20221219-1234.qcow2"
    file_name = "debian-11-genericcloud-amd64-20221219-1234.img"
    checksum  = "ba0237232247948abf7341a495dec009702809aa7782355a1b35c112e75cee81"
 }
}
resource "proxmox_virtual_environment_file" "cloud_config" {
  content_type = "snippets"
  datastore_id = "cephfs"
  node_name    = "proxmox1"

  source_file {
    path = "cloud-init/user-data.yml"
  }
}

virtual_machines.tf -->

resource "proxmox_virtual_environment_vm" "k8s_cp_01" {
  name        = "k8s-cp-01"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox1"
  vm_id       = "100"

  cpu {
    cores = 2
  }

  memory {
    dedicated = "4096"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "3A:53:4E:50:13:F6"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
     datastore_id      = "local-zfs"
     user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

     ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_01" {

  name        = "k8s-worker-01"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox1"
  vm_id       = "101"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "4096"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "2A:B4:2E:60:BD:5F"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_02" {

  name        = "k8s-worker-02"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox1"
  vm_id       = "102"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "4096"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "CA:1D:F9:41:75:BA"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_03" {

  name        = "k8s-worker-03"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox2"
  vm_id       = "103"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "6144"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "6A:EB:AB:27:EA:53"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_04" {

  name        = "k8s-worker-04"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox2"
  vm_id       = "104"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "6144"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "E6:3B:B6:D1:D4:2E"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_05" {

  name        = "k8s-worker-05"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox3"
  vm_id       = "105"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "6144"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "22:D1:D2:D0:51:35"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}
resource "proxmox_virtual_environment_vm" "k8s_worker_06" {

  name        = "k8s-worker-06"
  description = "Managed by Terraform"
  tags        = ["terraform"]
  node_name   = "proxmox3"
  vm_id       = "106"

  cpu {
    cores = 1
  }

  memory {
    dedicated = "6144"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge = "vmbr0"
    mac_address = "2E:B1:38:11:0E:A6"
  }

  disk {
    datastore_id = "local-zfs"
    file_id      = proxmox_virtual_environment_file.debian_cloud_image.id
    interface    = "scsi0"
    size         = "32"
  }

  serial_device {} # The Debian cloud image expects a serial port to be present

  operating_system {
    type = "l26" # Linux Kernel 2.6 - 5.X.
  }

  initialization {
    datastore_id      = "local-zfs"
    user_data_file_id = proxmox_virtual_environment_file.cloud_config.id

    ip_config {
      ipv4 {
        address = "dhcp"
      }
    }
  }
}

ansible.tf -->

resource "local_file" "ansible_inventory" {
  filename = "./ansible/inventory.ini"
  content     = <<-EOF
  k8s-cp-01 ansible_host=${proxmox_virtual_environment_vm.k8s_cp_01.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-01 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_01.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-02 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_02.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-03 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_03.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-04 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_04.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-05 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_05.ipv4_addresses[1][0]} ansible_become=true
  k8s-worker-06 ansible_host=${proxmox_virtual_environment_vm.k8s_worker_06.ipv4_addresses[1][0]} ansible_become=true

  [kube_control_plane]
  k8s-cp-01

  [etcd]
  k8s-cp-01

  [kube_node]
  k8s-worker-01
  k8s-worker-02
  k8s-worker-03
  k8s-worker-04
  k8s-worker-05
  k8s-worker-06

  [k8s_cluster:children]
  kube_node
  kube_control_plane

  EOF

  provisioner "local-exec" {
    command = "ansible-playbook -i inventory.ini playbook.yml"
    working_dir = "ansible"
    environment = {
      ANSIBLE_HOST_KEY_CHECKING = "false"
    }
  }
}

cloud-init/user-data.yml -->

#cloud-config
package_upgrade: true
packages: qemu-guest-agent

timezone: America/New_York
users:
  - default
  - name: adam
    ssh-authorized-keys:
      - ssh-rsa [ssh_public_key] adam@[some_machine_name]
    passwd: "[hashed_password_for_the_VM]"
    groups: [adm, cdrom, dip, plugdev, lxd, sudo]
    lock-passwd: false
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash

runcmd:
    - sed -i -e '/^PermitRootLogin/s/^.*$/PermitRootLogin yes/' /etc/ssh/sshd_config
    - sed -i -e '$aAllowUsers adam' /etc/ssh/sshd_config
    - sudo killall -HUP sshd
    - sudo apt-get update
    - sudo apt-get install open-iscsi -y

power_state:
    delay: now
    mode: reboot
    message: Rebooting after cloud-init completion
    condition: true

ansible/playbook.yml -->

- hosts: all
  become: yes
  tasks:
  - name: Set hostname from inventory
    hostname:
     name: "{{ inventory_hostname }}"
  vars:
    ansible_user: adam

# Import kubespray playbook to deploy k8s cluster
- import_playbook: ./kubespray/cluster.yml

ansible/ansible.cfg -->

[defaults]
timeout=60
deprecation_warnings=False
library       = ./library/:./kubespray/library/
roles_path    = ./roles/:./kubespray/roles/

This went through and created the VM's and then started the Ansible playbook to kick off a Kubespray install. Looks like the file_id's worked without pointing them anywhere since they are on shared storage.

Thanks for your help btw! It allowed me to debug this and get it working.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Need help in decreasing the complexity and amount of downloads for Ceph snippet/ISO storage for cloud-init based VM's... #678

{{title}}

Replies: 4 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Need help in decreasing the complexity and amount of downloads for Ceph snippet/ISO storage for cloud-init based VM's... #678

ab17182 Nov 3, 2023

Replies: 4 comments · 4 replies

otopetrik Nov 4, 2023

ab17182 Nov 4, 2023 Author

ab17182 Nov 4, 2023 Author

otopetrik Nov 4, 2023

ab17182 Nov 4, 2023 Author

ab17182 Nov 4, 2023 Author

ab17182 Nov 4, 2023 Author

ab17182 Nov 4, 2023 Author

ab17182
Nov 3, 2023

Replies: 4 comments 4 replies

otopetrik
Nov 4, 2023

ab17182
Nov 4, 2023
Author

ab17182 Nov 4, 2023
Author

ab17182 Nov 4, 2023
Author

ab17182 Nov 4, 2023
Author

ab17182
Nov 4, 2023
Author

ab17182
Nov 4, 2023
Author