Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Duplicated SSL config during multiple Kafka binders to different clusters (SSL secured and plaintext-open) #3068

Open
maistrovyi opened this issue Jan 17, 2025 · 0 comments

Comments

@maistrovyi
Copy link

maistrovyi commented Jan 17, 2025

Hey, faced with strange behavior that SSL config from kafka1 binder affects kafka2 binder.
For example, we are working with 2 different kafka clusters:

  • Kafka cluster №1 (let's say kafka-secured) that is secured with SSL
  • Kafka cluster №2 (let's say kafka-open) that is simply available via plaintext

Application.yml:

spring:
  application:
    name: local
  cloud:
    function:
      definition: securedConsumer;openConsumer
    stream:
      bindings:
        openConsumer-in-0:
          binder: kafka-open
          destination: "some-topic-in-kafka-open"
        securedConsumer-in-0:
          binder: kafka-secured
          destination: "some-topic-in-kafka-secured"
      binders:
        kafka-open:
          type: kstream
          default-candidate: false
          inherit-environment: false
          environment:
            spring.cloud.stream.kafka.streams:
              binder:
                brokers: "some-open-kafka-host:9092"
                configuration:
                  security.protocol: PLAINTEXT
              bindings:
                securedConsumer-in-0:
                  consumer:
                    startOffset: latest
                    application-id: "open-consumer-application-id"
                    key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
                    value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'
        kafka-secured:
          type: kstream
          default-candidate: false
          inherit-environment: false
          environment:
            spring.cloud.stream.kafka.streams:
              binder:
                brokers: "some-secured-kafka-host:9092"
                configuration:
                  security.protocol: SSL
                  ssl:
                    truststore:
                      location: "some.truststore.jks"
                      password: "some-pass"
                    keystore:
                      location: "some.keystore.jks"
                      password: "some-pass"
              bindings:
                openConsumer-in-0:
                  consumer:
                    startOffset: latest
                    application-id: "secured-consumer-application-id"
                    key-serde: 'org.apache.kafka.common.serialization.Serdes$VoidSerde'
                    value-serde: 'org.apache.kafka.common.serialization.Serdes$StringSerde'

Version of the framework
Tested both on spring-cloud-stream-binder-kafka-streams v4.2.0 and v3.2.10

Expected behavior
All streams from/to kafka-open working normally.

Actual behavior
Consumer is not subscribed to partitions and warns about:
org.apache.kafka.clients.NetworkClient : [Consumer clientId=..., groupId=...] Connection to node -3 (...) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.

In o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values: i see that open-consumer has SSL config from secured-consumer.
Probably, it's merged in org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsBinderSupportAutoConfiguration#streamConfigGlobalProperties

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant