[Bug]: renew not working with reverse-proxy and mtls=false

[juju4]

Steps to Reproduce

I setup a certificate server (step 0.28.2 on ubuntu 24.04.1) and a nginx reverse proxy.
Issuing certificate works fine from step service or nginx.
But renewal does not work when using nginx ca-url (port 443). It works if accessing directly step service (port 8443).
"mtls=false" was used. "--mtls false" as described in docs did not work with step-cli for me.
It does not seem there is any debug/verbose option to check where getting invalid character which is probably the start of a html page.

From cert-renewer systemd unit

ExecStart=/usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem --mtls false --force ${CERT_LOCATION} ${KEY_LOCATION} (code=exited, status=1/FAILURE)
# results in
Jan 08 21:50:43 myhost.internal step[3935]: too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'

Manual testing

root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem --mtls false /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'
root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value
root@myhost:~# /usr/bin/step ca renew --mtls false --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'
root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value

Thanks

Your Environment

• OS - Ubuntu 24.04.1
• step CLI Version - 0.28.2

Expected Behavior

Renewal to work

Actual Behavior

Renewal fails

Additional Context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[hslatman]

Hey @juju4,

Can you try --mtls=false in your manual testing (and cert-renewer systemd unit)? I believe the --mtls false gets interpreted differently than you expect.

As for the < character: it's possible that your proxy is serving an error page when trying to upstream to the CA server. I suppose because those invocations don't have --mtls=false, they actually are terminated by the mTLS endpoint, and a TLS error is returned. So I think if you provide --mtls=false, you won't get that error message. Alternatively, you could try inspecting the HTML in a browser.

[hslatman]

One location where --mtls false was being mentioned was updated in this PR: smallstep/docs#374.

[juju4]

I did use mtls=false too. forgot to include in the list and it returns same error "error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value".

I would want to debug with curl as /renew needs a POST but I don't know the syntax needed. Browser /renew returns blank page with HTTP 405 status code. while on above server logs a 400 status code.

# curl -X POST https://certs.internal/renew
{"status":400,"message":"The request could not be completed: missing client certificate."}
# curl -X POST https://certs.internal/renew -d @/etc/ssl/certs/myhost.crt 
{"status":400,"message":"The request could not be completed: missing client certificate."}