ACME: renewal fails #823

dwaldmannDE · 2022-02-10T10:21:00Z

dwaldmannDE
Feb 10, 2022

Hi!

I‘ve recently started testing with step-ca in my local environment and primarily use the ACME provisioner to get certificates for caddy webservers. Issuing the initial certificate works just fine, but the certificates are not renewed.

Here’s what the log of step-ca is telling me:

Feb 10 11:15:21 acme.internal.domain step-ca[497]: time="2022-02-10T11:15:21+01:00" level=info duration=4.789692ms duration-ns=4789692 fields.time="2022-02-10T11:15:21+01:00" method=POST name=ca nonce=eGwyRDdzRGk0N3hwMlJISmNWY0xReUptSlZTb2ZsMWo path=/acme/acme/authz/iJCpXpWmtLRTAHpD01fIvClDJJ6iKEOD protocol=HTTP/2.0 referer= remote-address=172.27.255.203 request-id=c82ebeb2k2bs4vfvlchg response="{"identifier":{"type":"dns","value":"server.internal.domain"},"status":"pending","challenges":[{"type":"dns-01","status":"pending","token":"SITIQwqsqVFAWUSp3ftfwsraIcX7qJ4I","url":"https://acme.internal.domain:8443/acme/acme/challenge/iJCpXpWmtLRTAHpD01fIvClDJJ6iKEOD/6EMT3wO08NDbUpziD1e53yutmCTdLvpz"},{"type":"http-01","status":"pending","token":"SITIQwqsqVFAWUSp3ftfwsraIcX7qJ4I","url":"https://acme.internal.domain:8443/acme/acme/challenge/iJCpXpWmtLRTAHpD01fIvClDJJ6iKEOD/oPI3KIkIoTi9XVv7eSziiaZpFfuZzdGv"},{"type":"tls-alpn-01","status":"pending","token":"SITIQwqsqVFAWUSp3ftfwsraIcX7qJ4I","url":"https://acme.internal.domain:8443/acme/acme/challenge/iJCpXpWmtLRTAHpD01fIvClDJJ6iKEOD/uQoKaqnrS12sxl1HLWzTQzDDUMNGdzqI","error":{"type":"urn:ietf:params:acme:error:connection","detail":"The server could not connect to validation target"}}],"wildcard":false,"expires":"2022-02-11T10:11:17Z"}" size=895 status=200 user-agent="Caddy/(devel) CertMagic acmez (linux; amd64)" user-id=

I think it the important part is this one: The server could not connect to validation target

The CA is running on acme.internal.domain and is listening to :8443 and there is the host trying to get the certificate renewed server.internal.domain. Name resolution works fine and I can curl from both machines to each other, too.

I probably messed up some configuration - but where to look for?

Installed versions:

step-ca 0.18.1
caddy 2.4.5

Answered by tashian

Feb 10, 2022

It's strange that you can curl from both machines to each other, and that the initial enrollment works.
My hunch is that it's not a DNS issue.

Do you get any errors or anything on the Caddy side?
I'm pretty sure the TLS-APLN-01 challenge type requires the ACME server to connect to the target on port 443. So, if your Caddy server for any reason is unable to listen on that port (eg lacking permissions, or a firewall, or another service using 443), it will not be able to renew the certificate.

You could also try configuring Caddy with a different challenge type and see if that works better.

View full answer

tashian · 2022-02-10T18:46:04Z

tashian
Feb 10, 2022
Collaborator

It's strange that you can curl from both machines to each other, and that the initial enrollment works.
My hunch is that it's not a DNS issue.

Do you get any errors or anything on the Caddy side?
I'm pretty sure the TLS-APLN-01 challenge type requires the ACME server to connect to the target on port 443. So, if your Caddy server for any reason is unable to listen on that port (eg lacking permissions, or a firewall, or another service using 443), it will not be able to renew the certificate.

You could also try configuring Caddy with a different challenge type and see if that works better.

7 replies

tashian Feb 12, 2022
Collaborator

Here's what seems to be happening:

Caddy is trying an http-01 challenge with step-ca.
30+ seconds later, the connection times out. Usually this sort of timeout happens because of a DNS or network configuration issue. In the past, I've seen this issue in Kubernetes clusters where the ACME client inside the cluster can reach a CA on the outside, but the CA can't resolve / reach the ACME client to verify the challenge.
Caddy gives up and tells the CA to stop trying. This is the "deactivating authorization" request. Caddy gets a response of "The request message was malformed." I'm not sure what is happening here. It may be a bug in step-ca, but it's not the core of the issue.

It's very strange to me that enrollment works but renewal doesn't.
Maybe it's an issue around the ACME account being persisted on the CA side.
Could you please post your ca.json and Caddyfile?
Also, I'd suggest looking through the Caddy forums for other folks who have encountered this issue, if you haven't already.

cc @mholt @hslatman

hslatman Feb 12, 2022
Collaborator

Will take a look at reproducing this, @tashian.

hslatman Feb 14, 2022
Collaborator

Tried reproducing this, but so far I'm not getting the same behavior as you, @dwaldmannDE 😞

I configured step-ca on acmeca.localhost:7443 with 1 minute certificate lifetime. For Caddy test.localhost as a host and on demand certificate creation/renewal for easier testing. Below is my Caddyfile:

{
	skip_install_trust
	debug
}

test.localhost:443 {
	tls {
		on_demand
		issuer acme {
			dir https://acmeca.localhost:7443/acme/acme20/directory
			trusted_roots /path/to/authority/certs/root_ca.crt
			disable_tlsalpn_challenge
		}
	}
	respond "Hello, World!"
}

When requesting test.localhost for the first time the cert is requested as usual. After cert expiration and opening a new connection to test.localhost, I see Caddy renewing the certificate:

2022/02/14 11:53:50.156	DEBUG	tls.handshake	matched certificate in cache	{"subjects": ["test.localhost"], "managed": true, "expiration": "2022/02/14 11:44:55.000", "hash": "f7f839ee37d094c376a31f03fd1f87702e45f46831cab1c824ecf0538d3f81e3"}
2022/02/14 11:53:50.156	INFO	tls.on_demand	attempting certificate renewal	{"server_name": "test.localhost", "subjects": ["test.localhost"], "expiration": "2022/02/14 11:44:55.000", "remaining": -535.156369}
2022/02/14 11:53:50.159	INFO	tls.renew	acquiring lock	{"identifier": "test.localhost"}
2022/02/14 11:53:50.184	INFO	tls.renew	lock acquired	{"identifier": "test.localhost"}
2022/02/14 11:53:50.184	INFO	tls.renew	renewing certificate	{"identifier": "test.localhost", "remaining": -535.184489}
...
2022/02/14 11:53:50.219	DEBUG	tls.issuance.acme.acme_client	no solver configured	{"challenge_type": "dns-01"}
2022/02/14 11:53:50.219	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "test.localhost", "challenge_type": "http-01", "ca": "https://acmeca.localhost:7443/acme/acme20/directory"}
2022/02/14 11:53:50.225	INFO	tls.issuance.acme	served key authentication	{"identifier": "test.localhost", "challenge": "http-01", "remote": "[::1]:55187", "distributed": false}
2022/02/14 11:53:50.226	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acmeca.localhost:7443/acme/acme20/challenge/Vx7jHt8Fegq8U7hDayhHsBSVLXa1XVFs/XxsdFZCwvfRwdFOCFsuGHq6YYqcPNMJB", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (darwin; arm64)"]}, "response_headers": {"Cache-Control":["no-store"],"Content-Length":["240"],"Content-Type":["application/json"],"Date":["Mon, 14 Feb 2022 11:53:50 GMT"],"Link":["<https://acmeca.localhost:7443/acme/acme20/directory>;rel=\"index\"","<https://acmeca.localhost:7443/acme/acme20/authz/Vx7jHt8Fegq8U7hDayhHsBSVLXa1XVFs>;rel=\"up\""],"Location":["https://acmeca.localhost:7443/acme/acme20/challenge/Vx7jHt8Fegq8U7hDayhHsBSVLXa1XVFs/XxsdFZCwvfRwdFOCFsuGHq6YYqcPNMJB"],"Replay-Nonce":["M2dhR0w1cFNjaXQ5MldUdVhyMDVLRzVZZ0FRSzlrTVc"]}, "status_code": 200}
2022/02/14 11:53:50.226	DEBUG	tls.issuance.acme.acme_client	challenge accepted	{"identifier": "test.localhost", "challenge_type": "http-01"}
2022/02/14 11:53:50.485	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acmeca.localhost:7443/acme/acme20/authz/Vx7jHt8Fegq8U7hDayhHsBSVLXa1XVFs", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (darwin; arm64)"]}, "response_headers": {"Cache-Control":["no-store"],"Content-Length":["794"],"Content-Type":["application/json"],"Date":["Mon, 14 Feb 2022 11:53:50 GMT"],"Link":["<https://acmeca.localhost:7443/acme/acme20/directory>;rel=\"index\""],"Location":["https://acmeca.localhost:7443/acme/acme20/authz/Vx7jHt8Fegq8U7hDayhHsBSVLXa1XVFs"],"Replay-Nonce":["OWpyRlhjMThBa2Jzem5ZTGZoR0ZzNEtBM2dKNWQ1UHc"]}, "status_code": 200}
2022/02/14 11:53:50.486	INFO	tls.issuance.acme.acme_client	validations succeeded; finalizing order	{"order": "https://acmeca.localhost:7443/acme/acme20/order/r2OyjS1O1z48eExYTDExyauN0iDd9Gk9"}
2022/02/14 11:53:50.492	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acmeca.localhost:7443/acme/acme20/order/r2OyjS1O1z48eExYTDExyauN0iDd9Gk9/finalize", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (darwin; arm64)"]}, "response_headers": {"Cache-Control":["no-store"],"Content-Length":["525"],"Content-Type":["application/json"],"Date":["Mon, 14 Feb 2022 11:53:50 GMT"],"Link":["<https://acmeca.localhost:7443/acme/acme20/directory>;rel=\"index\""],"Location":["https://acmeca.localhost:7443/acme/acme20/order/r2OyjS1O1z48eExYTDExyauN0iDd9Gk9"],"Replay-Nonce":["UVRGY001ejZQYjlSWVRWZDV5eDEyY0h5UUE5Y1Bkd2U"]}, "status_code": 200}
2022/02/14 11:53:50.495	DEBUG	tls.issuance.acme.acme_client	http request	{"method": "POST", "url": "https://acmeca.localhost:7443/acme/acme20/certificate/o8JJAevhYcyD3rmpoSz7sFQmIa4xwtfT", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.6 CertMagic acmez (darwin; arm64)"]}, "response_headers": {"Cache-Control":["no-store"],"Content-Length":["1466"],"Content-Type":["application/pem-certificate-chain; charset=utf-8"],"Date":["Mon, 14 Feb 2022 11:53:50 GMT"],"Link":["<https://acmeca.localhost:7443/acme/acme20/directory>;rel=\"index\""],"Replay-Nonce":["TjhGbXo2aEJ3NEVZZ1FlRWU3cGFZWmM1ZmV1ajRDbjQ"]}, "status_code": 200}
2022/02/14 11:53:50.495	INFO	tls.issuance.acme.acme_client	successfully downloaded available certificate chains	{"count": 1, "first_url": "https://acmeca.localhost:7443/acme/acme20/certificate/o8JJAevhYcyD3rmpoSz7sFQmIa4xwtfT"}
2022/02/14 11:53:50.498	INFO	tls.renew	certificate renewed successfully	{"identifier": "test.localhost"}
2022/02/14 11:53:50.498	INFO	tls.renew	releasing lock	{"identifier": "test.localhost"}

Of course this is all on my own machine and no additional network topology involved, so that might be different than your setup. What does yours look like? I saw the IP 172.27.255.203 being used, indicating a private network; are there DNS servers in the network?

As @tashian said: it's weird that the initial request for a certificate works and the renewals don't. In the and, the ACME flow is the same for both operations.

Regarding your question about the challenge types: clients are not leading in terms of what challenges they'd like to respond to. There's no way to do so in the ACME protocol as far as I know, although I admit that making the client choose up front does makes sense. When a new order is created, the server determines the challenge types and challenges to be solved. The client can choose to not support one of the proposed challenges; as long as it is able to solve one of them, the server authorizes the client. In this case disabling the tls-alpn-01 challenge in Caddy means that Caddy won't solve this type of challenge, even though the CA might respond with a challenge of this type.

@tashian: I think you're right about (3); from a cursory look it seems that we currently don't support the "Deactivating an Authorization" request: https://datatracker.ietf.org/doc/html/rfc8555#section-7.5.2.

dwaldmannDE Feb 14, 2022
Author

Thank you all for the participation and your suggestions.

TL;DR: I've set up a new instance of step-ca and this one is working fine. Issuing and renewal of certificates is working fine since Saturday evening. I'd assume something was broken with my original installation or things were messed up on vm level already.

This is the ca.json from the faulty instance:

{
"root": "/var/lib/step-ca/.step/certs/root_ca.crt",
"federatedRoots": null,
"crt": "/var/lib/step-ca/.step/certs/intermediate_ca.crt",
"key": "/var/lib/step-ca/.step/secrets/intermediate_ca_key",
"address": ":8443",
"insecureAddress": ":8080",
"dnsNames": [
"acme.internal.domain,
"alias.internal.domain",
"localhost"
],
"logger": {
"format": "text"
},
"db": {
"type": "badgerv2",
"dataSource": "/var/lib/step-ca/.step/db",
"badgerFileLoadingMode": ""
},
"authority": {
"provisioners": [
{
"type": "JWK",
"name": "[email protected]",
"key": {
"use": "sig",
"kty": "EC",
"kid": "c_ybMWoX6AnY2b3Us4gJEXmOVyb5Ts_ImnCvQdHDPxE",
"crv": "P-256",
"alg": "ES256",
"x": "IVewD0HJCnZcV9xPExr6UU6vZszyQ95XXtTdUYlf6_E",
"y": "XT3zIlEm0tTCR0j9MxC2g7KChvCisqbeFTJ2r1WFyDU"
},
"encryptedKey": "eyJhbGciOiJQQkVTMi1IUzI1NitBMTI4S1ciLCJjdHkiOiJqd2sranNvbiIsImVuYyI6IkEyNTZHQ00iLCJwMmMiOjEwMDAwMCwicDJzIjoieG9GWGtWQlQtUkpXQ0x6dXltanNFUSJ9.rB_ssGfaxxQlSVODSZKrHaKDL8roeX0aNi5s9aaDv0spXgJjAxgtnQ.qa4QNpQ2oenjS-a6.ZpB05BbN_3fZE5K4gXI2LOaq5v8CEXPRU-RqygNsQdzXx9T_fjYZzEYXc3RnBMMxks_1ovbhhnPh14vo6P_cbDsESlzb5bwUJwDMImxl9JpE5kuRSSgBqJwroitnzJwrK4oL5aYw325xprH8L24_3-Ra_hzPFfBuCMfIdbFE40xTyPWAOP-ZAdtR9yHOkzatyfaJYZQkIzh5-VyEvMHmWfSK58SNYQEvl_oO9PIn3QBl_0hpoXlAyH1N13v1LT4koYOJiZGQ1x5aOhR-dp-DZhgfRehup-fO-31SMXxoT1B7cvyKK3QAhNV2CYmzJll2N5BztGCZgXwzM0u37PM.pTCvOnVA1sLy1AAQDumQ-w"
},
{
"type": "ACME",
"name": "acme",
"forceCN": true,
"claims": {
"minTLSCertDuration": "5m",
"maxTLSCertDuration": "8h",
"defaultTLSCertDuration": "8h"
}
}
],
"template": {},
"backdate": "1m0s"
},
"tls": {
"cipherSuites": [
"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",
"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
],
"minVersion": 1.2,
"maxVersion": 1.3,
"renegotiation": false
}
}

This is one of the Caddyfiles being used:

server.internal.domain {
reverse_proxy 127.0.0.1:8080
tls {
issuer acme {
dir https://acme.internal.domain:8443/acme/acme/directory
email [email protected]
}
}
}

All servers are connected via a private network, which has a total of five local DNS-Servers.

tashian Feb 14, 2022
Collaborator

Glad to hear you got it working.

TheApproach · 2023-09-11T06:46:23Z

TheApproach
Sep 11, 2023

have run into the same issue on .private local URLs

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ACME: renewal fails #823

{{title}}

Replies: 2 comments 7 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

ACME: renewal fails #823

dwaldmannDE Feb 10, 2022

Replies: 2 comments · 7 replies

tashian Feb 10, 2022 Collaborator

tashian Feb 12, 2022 Collaborator

hslatman Feb 12, 2022 Collaborator

hslatman Feb 14, 2022 Collaborator

dwaldmannDE Feb 14, 2022 Author

tashian Feb 14, 2022 Collaborator

TheApproach Sep 11, 2023

dwaldmannDE
Feb 10, 2022

Replies: 2 comments 7 replies

tashian
Feb 10, 2022
Collaborator

tashian Feb 12, 2022
Collaborator

hslatman Feb 12, 2022
Collaborator

hslatman Feb 14, 2022
Collaborator

dwaldmannDE Feb 14, 2022
Author

tashian Feb 14, 2022
Collaborator

TheApproach
Sep 11, 2023