Unable to connect via secure tunnel when using authorized_keys and keyfile instead of password auth

[dwalkes]

Describe the bug

I've got a repo at https://github.com/Trellis-Logic/meta-aws-iot-demo which leverages the meta-aws layer for demonstrating secure tunneling and jobs features with AWS IoT on Yocto Poky distribution (currently using the kirkstone branch).

Everything works great with SSH connection through the procedure described at https://github.com/Trellis-Logic/meta-aws-iot-demo/wiki/Creating-a-Secure-Tunnel as long as I use username/password authentication.

When I attempt to switch to private key authentication and authorized_keys I am unable to connect to the device through the secure tunnel. I'm able to connect with the same private key over the local network without issue.

To Reproduce

• Start with https://github.com/Trellis-Logic/meta-aws-iot-demo and a raspberry pi target.
• Create a key pair using ssh-keygen and no password. I've tried -t rsa-sha2-512, -t rsa-sha2-256 or -t ed25519.
• Copy the .pub key to /home/root/.ssh/authorized_keys
• Verify ssh -i <private key> root@ip can connect successfully.
• Attempt to use the private key with Secure Tunneling and steps in https://github.com/Trellis-Logic/meta-aws-iot-demo/wiki/Creating-a-Secure-Tunnel instead of username and password authentication.

Expected behavior

I'm able to login with either password or private key authentication.

Actual behavior

The attempt to login with private key authentication fails with message "Failed to authenticate. Try again"

Logs

With verbose logging enabled in sshd (DEBUG3), I see this when I successfully login with the ssh key (over local network)

Apr 24 04:34:24 raspberrypi4-64 auth.info sshd[1854]: Accepted key RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E found at /home/root/.ssh/authorized_keys:1
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug2: check_authkeys_file: /home/root/.ssh/authorized_keys: processed 1/1 lines
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug1: restore_uid: 0/0
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_answer_keyallowed: publickey authentication: RSA key is allowed
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_send: entering, type 23
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_sshkey_verify: entering [preauth]
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_send: entering, type 24 [preauth]
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_receive_expect: entering, type 25 [preauth]
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_receive: entering [preauth]
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_receive: entering
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: monitor_read: checking request 24
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 verified
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug1: auth_activate_options: setting new authentication options
Apr 24 04:34:24 raspberrypi4-64 auth.debug sshd[1854]: debug3: mm_request_send: entering, type 25
Apr 24 04:34:24 raspberrypi4-64 auth.info sshd[1854]: Accepted publickey for root from 192.168.99.22 port 25994 ssh2: RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E

And I see this on the failure case when using AWS Secure Tunnel with the same private key

Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: trying public key file /home/root/.ssh/authorized_keys
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: fd 5 clearing O_NONBLOCK
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: /home/root/.ssh/authorized_keys:1: matching key found: RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: /home/root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Apr 24 04:24:27 raspberrypi4-64 auth.info sshd[1673]: Accepted key RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E found at /home/root/.ssh/authorized_keys:1
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug2: check_authkeys_file: /home/root/.ssh/authorized_keys: processed 1/1 lines
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: restore_uid: 0/0
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_answer_keyallowed: publickey authentication: RSA key is allowed
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_send: entering, type 23
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_sshkey_verify: entering [preauth]
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_send: entering, type 24 [preauth]
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_receive_expect: entering, type 25 [preauth]
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_receive: entering [preauth]
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_receive: entering
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: monitor_read: checking request 24
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug1: auth_activate_options: setting new authentication options
Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_request_send: entering, type 25
Apr 24 04:24:27 raspberrypi4-64 auth.info sshd[1673]: Failed publickey for root from 127.0.0.1 port 34720 ssh2: RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E

Difference appears to be:

Apr 24 04:24:27 raspberrypi4-64 auth.debug sshd[1673]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto

Environment (please complete the following information):

• OS: Yocto poky (https://github.com/Trellis-Logic/meta-aws-iot-demo)
• Version kirkstone
• Architecture: raspberrypi4-64 (same issue with RPI3)
• Device Client commit: cf76107 (from this line)

Thanks for reviewing!

[dwalkes]

I've tried the master branch of all repos as well, fails with the same message

debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto

Relevant releases of openssl/h on these branches:

• master
  • openssh is currently 9.7p1.
  • openssl is 3.2.1.
• kirkstone
  • openssh is 8.9p1.
  • openssl is 3.0.13.

Do you have any specific configurations you've successfully used for private key authentication? I could try to reproduce.

Other notes/observations:

• Similar description in the bug on publickey RSA signature unverified: error in libcrypto to RHEL9 sshd (with LEGACY crypto policy enabled) openssl/openssl#23513 which blames openssh
• Openssh error message looks like it comes from here.
• error in libcrypto text means SSH_ERR_LIBCRYPTO_ERROR is returned from sshkey_verify

[HarshGandhi-AWS]

Hello @dwalkes , thank you for sharing all of these details with us. You said you were able to connect to the device using the id/password but it is failing to connect over private key, can you tell us how did you create the private key? I am not sure if the private key you have generated is not working or it is an OpenSSH / OpenSSL issue.

[dwalkes]

Hi @HarshGandhi-AWS thanks for the response.

You said you were able to connect to the device using the id/password but it is failing to connect over private key, can you tell us how did you create the private key?

Yes see the detail in the first post:

Create a key pair using ssh-keygen and no password. I've tried -t rsa-sha2-512, -t rsa-sha2-256 or -t ed25519.

Regarding

I am not sure if the private key you have generated is not working or it is an OpenSSH / OpenSSL issue.

The key works for local access ssh -i <same private key I'm trying to use with AWS Remote Access Console> root@localdeviceip works, so I think the key is correct. Note in the traces I shared that the success case (from the local network) and the failure case (through aws-iot-device-client) both reference the same key SHA of RSA SHA256:KFmgrZ/JVrQAnUKqKXAZdSwcOupeMEYBN3a0aLo0F0E so I suspect this eliminates everything other than the connection between the aws-iot-device-client and OpenSSH/OpenSSL.

Can you answer this question?

Do you have any specific configurations you've successfully used for private key authentication? I could try to reproduce.

If you have a known working config and a specific verison of openssh/openssl tested I could try to reproduce with that version.

[HarshGandhi-AWS]

For testing purpose I use an ubuntu EC2 instance. The key generated by ec2 service is the private key I am using to connect to the destination device.

I tested device client again following these steps:

1. Created an new EC2 instance.
2. Installed and run the latest version of device client on the ec2 instance.
3. Created a new tunnel.
4. Used IoT console to connect to destination device using the private key generated by EC2.

I was able to connect to the device successfully and did not face any issue. I suspect either the issue in on:

1. Private key generated by you (which is not likely since you are able to ssh into your device using the same key as mentioned above).
2. The issue in on the yocto image.

To narrow down the issue, can you run device client on your destination device enabling secure tunneling feature and try to connect to it using the private key?

Would recommend running device client on a more powerful device than Raspberry pi to save time building client software locally. For testing locally, you can also run device client using our latest docker image hosted by us on this ECR repository. https://gallery.ecr.aws/aws-iot-device-client/aws-iot-device-client

[dwalkes]

@HarshGandhi-AWS thank again for the response.

Created an new EC2 instance.

Which OS? Which version of openssl/openssh is used? (try sshd --help or similar).

can you run device client on your destination device enabling secure tunneling feature and try to connect to it using the private key?

Sorry but I think that's exactly what I've done, or I don't understand what you are asking me to do differently than what I've already done. aws-iot-device-client is running on my destination device. Secure tunneling is enabled. I'm creating a secure tunnel and attempting to access it via the AWS Secure Tunnels page, providing a private key instead of a username/password.

Would recommend running device client on a more powerful device than Raspberry pi to save time building client software locally.

I have a cross toolchain for this through yocto so this isn't an issue.

[HarshGandhi-AWS]

Which OS? Which version of openssl/openssh is used?

OS: Ubuntu 22.04.4 LTS
OpenSSH: OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
OpenSSL: 3.0.2

Please be patient with us until we try to produce this issue locally and find its root cause. So far I am able to connect to destination device using the private key generated by the EC2 instance. I will try to create a new private key following the steps mentioned above in the issue detail and see if I can connect to the destination device using it or not.

Meanwhile, I would suggest using id/password for connecting to the device since it is working fine for you.

[dwalkes]

@HarshGandhi-AWS thanks for the response. I'll try to reproduce a successful connection to an AWS image using the same setup and compare logs. It looks like this configuration is very similar to the kirkstone configuration I have above.

I will try to create a new private key following the steps mentioned above in the issue detail and see if I can connect to the destination device using it or not

OK thanks.

Meanwhile, I would suggest using id/password for connecting to the device since it is working fine for you.

Yes I'm doing this, however it's not a suitable solution for production deployment.

Please be patient with us until we try to produce this issue locally and find its root cause.

You should also be able to reproduce if you build an image using https://github.com/Trellis-Logic/meta-aws-iot-demo/?tab=readme-ov-file#host-dependencies for RPI 3 or 4.

[HarshGandhi-AWS]

Hello @dwalkes , we were able to reproduce this issue locally. We suspect the issue is not the the device client itself but is on the web based secure tunneling which you were using to connect to the tunnel as a source. We are currently working with the console team to resolve this issue. Meanwhile if you want to connect to the device using private key you can use Local Proxy on your source device to connect to the tunnel. Team already tested it and it works fine when trying to connect as source using a device private key.

[aripanyvision]

Hi, same error trying to use pub/priv on a ubuntu desktop installed on a Jetson
local ssh with the private key works fine, in AWS IoT core web interface, tunneling works, user/pass auth works
but using the private key and the specific user im trying to login to, getting the same error as above no matter what i do

[dwalkes]

Any updates on the status of this one?

Meanwhile if you want to connect to the device using private key you can use Local Proxy on your source device to connect to the tunnel. Team already tested it and it works fine when trying to connect as source using a device private key.

I'm unable to use the local proxy either, using password access or private key. Can you share more detail about exactly how you are connecting?

I'm setting the key with

export AWSIOT_TUNNEL_ACCESS_TOKEN=<source key token value>

Then I'm starting the localproxy in source mode with -v 5 -r us-east-2 -s SSH=5555

Then attempting to connect with ssh -p 5555 root@localhost in another terminal window.

I get messages like this and the SSH command hangs without opening a terminal session.

[2024-06-14 04:51:59.915802] (0x00007fd09a945840) [info] creating tcp connection id 1
[2024-06-14 04:51:59.915845] (0x00007fd09a945840) [info] Accepted tcp connection on port 5555 from 127.0.0.1:55758
[2024-06-14 04:51:59.915923] (0x00007fd09a945840) [debug] Sending stream start, setting new stream ID to: 1, service id: SSH
[2024-06-14 04:51:59.916152] (0x00007fd09a945840) [debug] Starting web socket read loop while web socket is already reading. Ignoring...
[2024-06-14 04:51:59.916260] (0x00007fd09a945840) [debug] Prepare to send data message: service id: SSH stream id: 1 connection id: 1
[2024-06-14 04:51:59.916324] (0x00007fd09a945840) [debug] Write buffer has enough space, continue tcp read loop for SSH connection id: 1
[2024-06-14 04:51:59.916357] (0x00007fd09a945840) [debug] Not starting TCP read loop, socket is already reading
[2024-06-14 04:51:59.916369] (0x00007fd09a945840) [debug] not writing, no buffer contents, skip straight to being done draining

[dwalkes]

@HarshGandhi-AWS may I have an update on this please? It's blocking production deployment for us. Thanks!

[HarshGandhi-AWS]

Hello @dwalkes team is still working on this issue. Can you tell us why is this issue blocking your device deployment? The issue is on the cloud console and you can still use local proxy in source mode to connect to the device using the secure tunnel. That should be a good alternative for now. Can you use it to unblock yourself until our team works on fixing this console issue?

[dwalkes]

@HarshGandhi-AWS

you can still use local proxy in source mode to connect to the device using the secure tunnel

I wasn't able to get this to work either, as described at #455 (comment) - if you have further details about how you were able to make this work please share.

[RogerZhongAWS]

@dwalkes in your localproxy run command, try adding the arg --destination-client-type V1 for connecting to clients using an older protocol, which device client is currently.

ref: https://github.com/aws-samples/aws-iot-securetunneling-localproxy/releases/tag/v3.1.2

[dwalkes]

@RogerZhongAWS thanks that helps, I can confirm I'm able to use this command to start the proxy:

export AWSIOT_TUNNEL_ACCESS_TOKEN=<source access token from tunnel setup>
export AWS_REGION=<my_region>
docker run --rm -it -p 5555:5555 -e AWSIOT_TUNNEL_ACCESS_TOKEN=${AWSIOT_TUNNEL_ACCESS_TOKEN} public.ecr.aws/aws-iot-securetunneling-localproxy/ubuntu-bin:amd64-latest -v 5 -r ${AWS_REGION} -s SSH=5555 --destination-client-type V1 -b "*"

and then from the same host, I can use this command to reach the remote client

ssh -i <path to key> -p 5555 root@localhost

So this is in fact a workaround for me until the web console is fixed.

[HarshGandhi-AWS]

Hello @dwalkes, I believe this issue is now resolved and you can use IoT Console with certificates to connect to your device over secure tunnel. Can you please verify and let us know is this issue can be closed now?

[dwalkes]

@HarshGandhi-AWS same error unfortunately, traces look the same as initially shared in #455 (comment) with error message

Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto

I'm curious if you could share the debug3 messages from sshd on a working case for comparision.

I'm setting /etc/ssh/sshd_config with LogLevel DEBUG3

Result in the faillure case looks like this:

tail -f /var/log/messages
...
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: userauth_pubkey: publickey have rsa-sha2-512 signature for RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_key_allowed: entering [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_send: entering, type 22 [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive_expect: entering, type 23 [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive: entering [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive: entering
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: monitor_read: checking request 22
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_answer_keyallowed: entering
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: temporarily_use_uid: 0/0 (e=0/0)
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: trying public key file /home/root/.ssh/authorized_keys
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: fd 5 clearing O_NONBLOCK
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: /home/root/.ssh/authorized_keys:1: matching key found: RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: /home/root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Jul 13 15:51:17 client-proj auth.info sshd[7938]: Accepted key RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ found at /home/root/.ssh/authorized_keys:1
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug2: check_authkeys_file: /home/root/.ssh/authorized_keys: processed 1/1 lines
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: restore_uid: 0/0
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_answer_keyallowed: publickey authentication: RSA key is allowed
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_send: entering, type 23
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_sshkey_verify: entering [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_send: entering, type 24 [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive_expect: entering, type 25 [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive: entering [preauth]
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_receive: entering
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: monitor_read: checking request 24
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug1: auth_activate_options: setting new authentication options
Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_request_send: entering, type 25
Jul 13 15:51:17 client-proj auth.info sshd[7938]: Failed publickey for root from 127.0.0.1 port 44124 ssh2: RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ

Result in success case (ssh from the same network with the same key)

Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: userauth_pubkey: publickey have rsa-sha2-512 signature for RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_key_allowed: entering [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_send: entering, type 22 [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive_expect: entering, type 23 [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive: entering [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive: entering
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: monitor_read: checking request 22
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_answer_keyallowed: entering
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: temporarily_use_uid: 0/0 (e=0/0)
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: trying public key file /home/root/.ssh/authorized_keys
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: fd 5 clearing O_NONBLOCK
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: /home/root/.ssh/authorized_keys:1: matching key found: RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: /home/root/.ssh/authorized_keys:1: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Jul 13 15:49:46 client-proj auth.info sshd[6256]: Accepted key RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ found at /home/root/.ssh/authorized_keys:1
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug2: check_authkeys_file: /home/root/.ssh/authorized_keys: processed 1/1 lines
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: restore_uid: 0/0
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_answer_keyallowed: publickey authentication: RSA key is allowed
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_send: entering, type 23
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_sshkey_verify: entering [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_send: entering, type 24 [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_sshkey_verify: waiting for MONITOR_ANS_KEYVERIFY [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive_expect: entering, type 25 [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive: entering [preauth]
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_receive: entering
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: monitor_read: checking request 24
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 verified
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug1: auth_activate_options: setting new authentication options
Jul 13 15:49:46 client-proj auth.debug sshd[6256]: debug3: mm_request_send: entering, type 25
Jul 13 15:49:46 client-proj auth.info sshd[6256]: Accepted publickey for root from 192.168.99.30 port 44490 ssh2: RSA SHA256:gqXYikBDMPEfBEBRfVZP2BQrXRFf91E9pJ8fejK44hQ

Note the difference in the failure case, same as before:

Jul 13 15:51:17 client-proj auth.debug sshd[7938]: debug3: mm_answer_keyverify: publickey RSA signature using rsa-sha2-512 unverified: error in libcrypto

If you have the details about the ticket which was fixed in the web console I'd like to review the fix to try to understand what would have caused the libcrypto error previously and how this was resolved.

[HarshGandhi-AWS]

@dwalkes Thank you for letting us know. I will escalate the issue to resolve this as soon as possible. Sorry for the inconvenience.

[HarshGandhi-AWS]

Hello @dwalkes , upon speaking with console team we found that only PEM formatted (256 & 512) RSA keys are working with IoT Secure Tunneling console. Can you create the key in PEM format and retry?

ssh-keygen -m PEM -t rsa -b 4096

[dwalkes]

Hi @HarshGandhi-AWS thanks I can confirm this fixes the issue.

I can also continue to use my existing key after converting to pem with

ssh-keygen -p -f ~/.ssh/original-key -m pem

[kkurczewski]

@HarshGandhi-AWS

Hi, indeed RSA PEM is working, but shall we update documentation to reflect current state? Currently it says:

To use the browser-based SSH

[...] If you're authenticating using the private key, you can use RSA, DSA, ECDSA (nistp-*) and ED25519 key types, in PEM (PKCS#1, PKCS#8) and OpenSSH formats.

While I believe ssh-keygen generates open ssh key and it doesn't work.

[HarshGandhi-AWS]

Sure. We will work on updating the documentation. Thank you for confirming the fix is working for you.
cc: @ig15

[ig15]

Acknowledged, will add it in the next update. Thanks.