Error during challenge, nonce not found. #1367
Replies: 7 comments 3 replies
-
|
Hey @MauriceMossIT, I noticed that in the log for the renew, there's also an error: INFO[1339] duration=51.67783ms duration-ns=51677830 fields.time="2023-04-28T19:35:44Z" method=POST name=ca nonce=SHp6MmtoWmc4SDNLYVBWNkpQUjFJZjVlRWNTcjNwb00 path=/acme/acme/authz/CfIFlTUrP29ACUwQEYolv7pnXGqs5Z8p protocol=HTTP/1.1 referer= remote-address=10.6.18.5 request-id=ch620474b3p3d8hjq3k0 response="{\"identifier\":{\"type\":\"dns\",\"value\":\"test.com\"},\"status\":\"pending\",\"challenges\":[{\"type\":\"dns-01\",\"status\":\"pending\",\"token\":\"PeOCnWKydkoWVkl9zjj9XmJLToiGUbOj\",\"url\":\"https://10.6.18.10:8443/acme/acme/challenge/CfIFlTUrP29ACUwQEYolv7pnXGqs5Z8p/4fiqyBM9SWnHXZdVETTWlVIsjlCd8SIh\"},{\"type\":\"http-01\",\"status\":\"pending\",\"token\":\"PeOCnWKydkoWVkl9zjj9XmJLToiGUbOj\",\"url\":\"https://10.6.18.10:8443/acme/acme/challenge/CfIFlTUrP29ACUwQEYolv7pnXGqs5Z8p/8wCRyrKI7wDCvaUzBoP1dKhh4nC2lwV0\"},{\"type\":\"tls-alpn-01\",\"status\":\"pending\",\"token\":\"PeOCnWKydkoWVkl9zjj9XmJLToiGUbOj\",\"url\":\"https://10.6.18.10:8443/acme/acme/challenge/CfIFlTUrP29ACUwQEYolv7pnXGqs5Z8p/rC3UNHECEANsWQglaexwf7ym2nBQc8ds\"}],\"wildcard\":false,\"expires\":\"2023-04-29T19:35:44Z\"}" size=733 status=200 user-agent="A10 ACME client" user-id=So, the root cause of your error is the client trying to reuse the nonce. It should ensure to use a fresh nonce. Ideally, it would automatically detect this specific error, retrieve a new nonce, and continue working. I'm not 100% sure, but I think the CA already sends a new nonce in the error response sent back to the client, also in this specific case, which can then be used for a subsequent request automatically, without the client having to call the However, it's weird that it works the first time, because the same |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hslatman, I tested running a constant curl to port 80 from a separate device and can confirm port 80 is not closing. What's strange is this same load balancer config with a public domain can run against the let's encrypt staging or public servers without issue. The only issue is when it runs against Step-CA. Of course, a separate linux machine doesn't have issues with Step-CA. So it's hard to determine where the fault lies since both the load balancer and Step-CA work fine but not when used together. Is there any way to see the nonce sent in a packet capture? Is this in the encrypted traffic? If so I can try to decrypt the traffic and analyze it to confirm if a duplicate nonce is sent. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the nonce is indeed within the encrypted traffic. The CA will put the next nonce to use in the HTTP You mention that you don't see this issue with other ACME servers, so you don't see the request timeout in those cases? It's not the root cause of the nonce reuse, but it's still interesting that it times out on a renew; not on the first issuance. |
Beta Was this translation helpful? Give feedback.
-
|
@MauriceMossIT did you find a way to capture the TLS session keys on your load balancer? In the meantime I've discussed adding capturing the keys on the CA side with the team. While considered useful, the suggestion was to add full request/response logging instead. This should provide you with the same view of what's going on, but without having to start a packet dump and decrypting TLS. This is something I consider useful in more debugging scenarios, so I might do a quick implementation of that. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hslatman, Thank you for the response and consideration of further logging on the CA side. I do believe that would be very useful. Unfortunately I have not found any way to capture the session key on the load balancer. I have discovered that on an older version of the load balancer code this issue doesn't occur. I have reached out for the load balancer support to investigate this further, but nothing has been found yet. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hslatman, Load balancer support team advised the change in versions should have only had an upgrade of the acme.sh script from 2.8.6 to 3.0.1. Are you aware of any issues specific to version 3.0.1? |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hslatman, Just wanted to check and see if you had further considered implementing further logging. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @MauriceMossIT, sorry for the silence. It's still on my list to improve the logs, but some other things have higher priority at the moment. Hope to get to it soon. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for the update. I completely understand the delay. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @hslatman, I wanted to check in again and see if there's been any updates to the logging? I tested this issue again with some newer code on the load balancer, but no change when using Step-CA. |
Beta Was this translation helpful? Give feedback.
-
Hello,
I have a Step-CA server that I am able to use with a linux machine in standalone mode and enroll/renew without issue. However, I also have a load balancer that can respond to http-01 challenges to perform ACME. When using this load balancer my Step-CA server errors after the first enrollment or first renewal attempt. For example, if a cert renews successfully then tries to renew again later, I get an error. I also get this error if I try to enroll then renew, enroll succeeds then renew fails. Once I restart the Step-CA server, I can have 1 enrollment or 1 renewal succeed before further enrollments/renewals fail.
Server side error:
Client side error:
Does anyone have any idea why this issue might occur on the load balancer? If it is something with the nonce, is there a way to see that? Otherwise what else could I look at? A packet capture wasn't the most useful since most of the communication was encrypted. However, I do notice on successful runs the load balancer immediately responds to the first GET for the challenge with a 200 OK. On unsuccessful runs the load balancer doesn't respond to the first GET but does respond to a 2nd GET. I am unsure of why this occurs though.
Full server and client logs below. These logs were from a successful enrollment followed by the first renewal which failed.
Server
Client
Beta Was this translation helpful? Give feedback.
All reactions