keyfile query parameter doesn't work in connection string when ~/.ssh/config is empty

[tlhakhan]

System Information

Linux distribution

Debian GNU/Linux 12

Terraform version

$ terraform -v
Terraform v1.9.6
on linux_arm64
+ provider registry.terraform.io/dmacvicar/libvirt v0.8.0

Provider and libvirt versions

$ terraform providers 

Providers required by configuration:
.
└── provider[registry.terraform.io/dmacvicar/libvirt] 0.8.0

---

Description of Issue/Question

I observed the keyfile query param is ignored, and only the ~/.ssh/id_rsa is added to the list of key files, when ~/.ssh/config is an empty file.

I expect the keyfile provided in the query param to be included in the list of key files, even when ~/.ssh/config file is just an empty file.

Problem Setup

The system setup:

• The ~/.ssh/config is an empty file. The file must exist.
• Create an SSH private key ansible.key in the same folder as the Terraform workspace.
• A remote host running libvirtd.

The main.tf file:

terraform {
  required_providers {
    libvirt = {
      source  = "dmacvicar/libvirt"
      version = "0.8.0"
    }
  }
  backend "local" {
    path = "local.tfstate"
  }
}

provider "libvirt" {
  uri = "qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1"
}

data "libvirt_node_info" "node" {}

Steps to Reproduce Issue

Run terraform plan with the above system setup.

$ tf plan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
│ 
│   with provider["registry.terraform.io/dmacvicar/libvirt"],
│   on main.tf line 13, in provider "libvirt":
│   13: provider "libvirt" {
│ 
╵

Relevant logs

• Ran the following TF_LOG=debug tf plan
• Extracted the useful log lines below

2024-09-29T11:53:10.625Z [DEBUG] provider.terraform-provider-libvirt_v0.8.0: plugin address: address=/tmp/plugin103822099 network=unix timestamp=2024-09-29T11:53:10.625Z
2024-09-29T11:53:10.636Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] Configuring provider for 'qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1': &{map[uri:0x40002a6dc0] 0x40000a8960 <nil> 0x40000c3180 map[] <nil> {{<nil>} <nil>} 0x400009c400 0x400000c5b8 0x40000fa680 false {{{} 1} {0 0}} false false}: timestamp=2024-09-29T11:53:10.636Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [INFO] establishing ssh connection to 'vhost-1.lan': timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] auth methods for vhost-1.lan: privkey: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] found no ssh keys, using default keypath: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] ssh identity files for host 'vhost-1.lan': [${HOME}/.ssh/id_rsa]: timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [DEBUG] Reading ssh key '${HOME}/.ssh/id_rsa': timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.637Z [INFO]  provider.terraform-provider-libvirt_v0.8.0: 2024/09/29 11:53:10 [INFO] SSH connecting to 'vhost-1.lan' (vhost-1.lan): timestamp=2024-09-29T11:53:10.637Z
2024-09-29T11:53:10.792Z [ERROR] provider.terraform-provider-libvirt_v0.8.0: Response contains error diagnostic: @module=sdk.proto diagnostic_summary="failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain" tf_req_id=3cc81bda-7e2c-db96-3131-50901aeb5b17 @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.6 tf_provider_addr=provider tf_rpc=Configure timestamp=2024-09-29T11:53:10.792Z
2024-09-29T11:53:10.792Z [ERROR] vertex "provider[\"registry.terraform.io/dmacvicar/libvirt\"]" error: failed to connect: failed to connect to remote host 'vhost-1.lan': ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

---

Relevant code section:

The bug appears to be here in this code block (v0.8.0).

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 37 to 59 in 1a88931

	// keyfile order of precedence:
	// 1. load uri encoded keyfile
	// 2. load override as specified in ssh config
	// 3. load default ssh keyfile path
	sshKeyPaths := []string{}
	sshKeyPath := q.Get("keyfile")
	if sshKeyPath != "" {
	sshKeyPaths = append(sshKeyPaths, sshKeyPath)
	}
	keyPaths, err := sshcfg.GetAll(target, "IdentityFile")
	if err != nil {
	log.Printf("[WARN] unable to get IdentityFile values - ignoring")
	} else {
	sshKeyPaths = append(sshKeyPaths, keyPaths...)
	}
	if len(keyPaths) == 0 {
	log.Printf("[DEBUG] found no ssh keys, using default keypath")
	sshKeyPaths = []string{defaultSSHKeyPath}
	}
	log.Printf("[DEBUG] ssh identity files for host '%s': %s", target, sshKeyPaths)

Connection string for reference:

provider "libvirt" {
  uri = "qemu+ssh://[email protected]/system?keyfile=ansible.key&sshauth=privkey&no_verify=1"
}

In this section, the keyfile query param value in the connection string is retrieved and added to sshKeyPaths.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 42 to 45 in 1a88931

	sshKeyPath := q.Get("keyfile")
	if sshKeyPath != "" {
	sshKeyPaths = append(sshKeyPaths, sshKeyPath)
	}

In this section, the ~/.ssh/config file is scanned for any instance of IdentityFile.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 47 to 52 in 1a88931

	keyPaths, err := sshcfg.GetAll(target, "IdentityFile")
	if err != nil {
	log.Printf("[WARN] unable to get IdentityFile values - ignoring")
	} else {
	sshKeyPaths = append(sshKeyPaths, keyPaths...)
	}

In this section, the keyfile query param value is overwritten because ~/.ssh/config file is empty and then initializes with the defaultSSHKeyPath which is just ${HOME}/.ssh/id_rsa.

terraform-provider-libvirt/libvirt/uri/ssh.go

Lines 54 to 57 in 1a88931

	if len(keyPaths) == 0 {
	log.Printf("[DEBUG] found no ssh keys, using default keypath")
	sshKeyPaths = []string{defaultSSHKeyPath}
	}

Workaround

With the above understanding of the bug, there is one happy path. If the ~/.ssh/config includes any instance of IdentityFile for the given host, then it will also accept the query param keyfile.

# cat ~/.ssh/config
Host vhost-1.lan
  IdentityFile ~/.ssh/some-random.key

In the terraform log below, see that it reads the ansible.key from the connection string, and the ~/.ssh/some-random.key from the ~/.ssh/config. It then connects to vhost-1 successfully.

[INFO] establishing ssh connection to 'vhost-1.lan': timestamp=2024-09-29T12:07:30.325Z
[DEBUG] auth methods for vhost-1.lan: privkey: timestamp=2024-09-29T12:07:30.325Z
[DEBUG] ssh identity files for host 'vhost-1.lan': [ansible.key ~/.ssh/some-random.key]: timestamp=2024-09-29T12:07:30.325Z
[DEBUG] Reading ssh key 'ansible.key': timestamp=2024-09-29T12:07:30.325Z
[DEBUG] Reading ssh key '~/.ssh/some-random.key': timestamp=2024-09-29T12:07:30.325Z
[ERROR] Failed to parse ssh key: ssh: no key found: timestamp=2024-09-29T12:07:30.326Z
[INFO] SSH connecting to 'vhost-1.lan' (vhost-1.lan): timestamp=2024-09-29T12:07:30.326Z
[INFO] libvirt client libvirt version: 10000000: timestamp=2024-09-29T12:07:30.524Z

$ tf plan
data.libvirt_node_info.node: Reading...
data.libvirt_node_info.node: Read complete after 0s [id=2357352559]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

[dmacvicar]

@memetb