TLS connection to MSK brokers: org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

[hervekhg]

Hello,
I'm trying to Setup AKHQ with TLS connection to MSK but it's not working. I have this error message : Failed to create new KafkaAdminClient

My AKHQ TLS Config

  connections:
    test:
      properties:
        bootstrap.servers: "b-1.test-kafka-ms.xxxxxx.c3.kafka.eu-west-3.amazonaws.com:9094,b-3.test-kafka-ms.xxxxc3.kafka.eu-west-3.amazonaws.com:9094,b-2.tels-kafka-ms.xxxxx.c3.kafka.eu-west-3.amazonaws.com:9094"
        security.protocol: "SSL"
        ssl.protocol: "TLS"
        ssl.truststore.location: "/tmp/akhq/keystore.jks"
        ssl.truststore.password: "tmp-akhq-tls"
        ssl.truststore.type: "JKS"
        ssl.keystore.type: "PKCS12"
        ssl.keystore.location: /tmp/akhq/keymsk.p12
        ssl.keystore.password: "tmp-akhq-tls"
        ssl.key.password: "tmp-akhq-tls

Script used to generate keystore

CERT_DIR="/tmp/akhq"
SERVICE="tmp-akhq-tls"

openssl pkcs12 -export \
  -inkey ${CERT_DIR}/keymsk.pem \
  -in ${CERT_DIR}/certmsk.pem \
  -out ${CERT_DIR}/keymsk.p12 \
  -passout pass:${SERVICE} \
  -name akhq


keytool -v -importkeystore \
  -noprompt \
  -srckeystore ${CERT_DIR}/keymsk.p12 \
  -srcstoretype PKCS12 \
  -srcstorepass ${SERVICE} \
  -destkeystore ${CERT_DIR}/keystore.jks  \
  -deststoretype JKS \
  -storepass ${SERVICE} \
  -keypass ${SERVICE}

Error Message

2021-10-21 08:13:46,725 ERROR inclient-2 o.a.k.c.NetworkClient      [AdminClient clientId=adminclient-2] Connection to node -3 (b-2.int-infra-kafka-tls-ms.1rumhb.c3.kafka.eu-west-3.amazonaws.com/10.194.161.38:9094) failed authentication due to: SSL handshake failed
2021-10-21 08:13:46,726 WARN  inclient-2 c.a.i.AdminMetadataManager [AdminClient clientId=adminclient-2] Metadata update failed due to authentication error
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(Unknown Source)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown Source)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1333)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1264)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(Unknown Source)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(Unknown Source)
        at java.base/sun.security.validator.Validator.validate(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown Source)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(Unknown Source)
        ... 20 common frames omitted
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(Unknown Source)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown Source)
        at java.base/java.security.cert.CertPathBuilder.build(Unknown Source)
        ... 26 common frames omitted
2021-10-21 08:13:46,809 ERROR inclient-1 o.a.k.c.NetworkClient      [AdminClient clientId=adminclient-1] Connection to node -3 (b-2.int-infra-kafka-tls-ms.1rumhb.c3.kafka.eu-west-3.amazonaws.com/10.194.161.38:9094) failed authentication due to: SSL handshake failed
2021-10-21 08:13:46,810 WARN  inclient-1 c.a.i.AdminMetadataManager [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed

[tchiotludo]

The error is explicit 😄
java.nio.file.NoSuchFileException: /tmp/akhq/keymsk.p12, the akhq server can't find the file because it don't exist

[tchiotludo]

bette one, seems that your jks for CA is invalid.
Try to generate a simple one, like this : https://help.aiven.io/en/articles/5241419-configuring-java-ssl-to-access-kafka

[hervekhg]

Exactly the same error :/

[tchiotludo]

have you tried to generate the both cert & trustore ?
The error seems to mean that the broker url is not on the certificate, maybe open the certificate to see what url is inside ?

[w4rppy]

Hi,
You can remove:

ssl.protocol: "TLS"
ssl.truststore.location: "/tmp/akhq/keystore.jks"
ssl.truststore.password: "tmp-akhq-tls"
ssl.truststore.type: "JKS"

And it should work 👍

[hervekhg]

Thanks @w4rppy
The right conf.

connections:
    test:
      properties:
        bootstrap.servers: "b-1.test-kafka-ms.xxxxxx.c3.kafka.eu-west-3.amazonaws.com:9094,b-3.test-kafka-ms.xxxxc3.kafka.eu-west-3.amazonaws.com:9094,b-2.tels-kafka-ms.xxxxx.c3.kafka.eu-west-3.amazonaws.com:9094"
        security.protocol: "SSL"
        ssl.keystore.type: "PKCS12"
        ssl.keystore.location: /tmp/akhq/keymsk.p12
        ssl.keystore.password: "tmp-akhq-tls"
        ssl.key.password: "tmp-akhq-tls