STORM-4075 Supprt mTLS between Storm and ZK

[purushah]

What is the purpose of the change

(Explain why we should have this change)

How was the change tested

(Explain what tests did you do to verify the code change)

[purushah]

I successfully tested this setup locally with ZooKeeper (ZK) running with mTLS enabled.

ZooKeeper Configuration

client.certReload=true
secureClientPort=50513
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory

clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
ssl.keyStore.location=/tmp/ssl/server.keystore.jks
ssl.keyStore.password=password
ssl.trustStore.location=/tmp/ssl/server.truststore.jks
ssl.trustStore.password=password
ssl.hostnameVerification=false
ssl.clientAuth=need

Storm Configuration

storm.zookeeper.ssl.enable: true
storm.zookeeper.ssl.keystore.path: /tmp/ssl/client.keystore.jks
storm.zookeeper.ssl.keystore.password: password
storm.zookeeper.ssl.truststore.path:/tmp/ssl/client.truststore.jks
storm.zookeeper.ssl.truststore.password: password
storm.zookeeper.ssl.hostnameVerification: false

storm.zookeeper.servers:
  - "127.0.0.1"
storm.zookeeper.port: 50513

When ZK SSL is enabled, and storm.zookeeper.ssl.enable: false

2024-09-26 14:24:28,430 [myid:] - ERROR [nioEventLoopGroup-7-6:o.a.z.s.NettyServerCnxnFactory$CertificateVerifier@466] - Unsuccessful handshake with session 0x0
2024-09-26 14:24:28,430 [myid:] - WARN  [nioEventLoopGroup-7-6:o.a.z.s.NettyServerCnxnFactory$CnxnChannelHandler@302] - Exception caught
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d00000000000000000000000000004e200000000000000000000000100000000000000000000000000000000000
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 0000002d00000000000000000000000000004e200000000000000000000000100000000000000000000000000000000000
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1314)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        ... 17 common frames omitted
2024-09-26 14:24:30,009 [myid:] - ERROR [nioEventLoopGroup-7-7:o.a.z.s.NettyServerCnxnFactory$CertificateVerifier@466] - Unsuccessful handshake with session 0x0

When ZK SSL is enabled, and storm.zookeeper.ssl.enable: true, and all other settings are configured correctly.

2024-09-26 14:25:03,784 [myid:] - INFO  [nioEventLoopGroup-7-8:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:03,944 [myid:] - INFO  [nioEventLoopGroup-7-10:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:03,979 [myid:] - INFO  [nioEventLoopGroup-7-9:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:04,107 [myid:] - INFO  [nioEventLoopGroup-7-11:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:04,140 [myid:] - INFO  [nioEventLoopGroup-7-13:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:04,172 [myid:] - INFO  [nioEventLoopGroup-7-12:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:04,259 [myid:] - INFO  [nioEventLoopGroup-7-14:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'
2024-09-26 14:25:04,288 [myid:] - INFO  [nioEventLoopGroup-7-15:o.a.z.s.a.X509AuthenticationProvider@271] - Authenticated Id 'CN=Client,OU=IT,O=YourOrganization,L=YourCity,ST=YourState,C=YourCountry' for scheme 'x509'

[snikhil5]

lgtm. We need to update the documents too https://storm.apache.org/releases/current/SECURITY

[purushah]

Thanks, @snikhil5 will update the doc as part of https://issues.apache.org/jira/browse/STORM-4079