[Bug]: kafka: cannot connect to TLS kafka with TLS + plaintext

[edefaria]

What happened?

Using v1.66.0
The PR 6270, change the way to load TLS on kafka.
Previously with kafka.[producer,consumer].authentication=plaintext if kafka.[producer,consumer].tls.enabled, || config.TLS.Enabled was loading setTLSConfiguration.

The new code only loads TLS when authentication=tls, failing the connection with other authentication (plaintext/kerberos) + TLS (does not upgrade to TLS during authentication).
Basically it has become impossible to configure SASL-SSL with PLAIN.

Steps to reproduce

1. Run jaeger-collector/jaeger-ingester with kafka + TLS producer/consumer enabled and authentification = plaintext (kerberos is probably also impacted).
2. Crash to connect to remote kafka.
3. See Relevant log output

Expected behavior

It should work as before.

Relevant log output

Feb 17 15:47:19 infra-2.local docker-compose[3961324]: jaeger-collector_1  | {"level":"fatal","ts":1739807239.7673259,"caller":"collector/main.go:75","msg":"Failed to init storage factory","error":"kafka: client has run out of available brokers to talk to: 5 errors occurred:\n\t* EOF\n\t* unexpected EOF\n\t* dial tcp: address broker-004.local:9091broker-005.local:9091: too many colons in address\n\t* unexpected EOF\n\t* unexpected EOF\n","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:75\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:1041\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/collector/main.go:152\nruntime.main\n\truntime/proc.go:272"}

Feb 17 15:49:08 infra-2.local docker-compose[3964107]: jaeger-ingester_1   | {"level":"fatal","ts":1739807348.9089887,"caller":"ingester/main.go:72","msg":"Unable to create consumer","error":"kafka: client has run out of available brokers to talk to: 5 errors occurred:\n\t* unexpected EOF\n\t* unexpected EOF\n\t* EOF\n\t* dial tcp: address broker-004.local:9091broker-005.local:9091: too many colons in address\n\t* unexpected EOF\n","stacktrace":"main.main.func1\n\tgithub.com/jaegertracing/jaeger/cmd/ingester/main.go:72\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:985\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:1117\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:1041\nmain.main\n\tgithub.com/jaegertracing/jaeger/cmd/ingester/main.go:108\nruntime.main\n\truntime/proc.go:272"}

Screenshot

No response

Additional context

No response

Jaeger backend version

v1.66.0

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

No response

Deployment model

No response

Deployment configs

[sans-sehgal]

New here - picking this up.

[jkowall]

What instrumentation are you using for this setup?

[edefaria]

What instrumentation are you using for this setup?

The kafka server is configured to work with SASL_SSL (only SSL/TLS listener are available with a signed certificate) with PLAIN authentication.

My Kafka settings for Jaeger regarding the collector (same for ingester with consumer) are as follows:

--kafka.producer.brokers=broker-001...:9091,....
--kafka.producer.topic=...
--kafka.producer.authentication=plaintext
--kafka.producer.tls.enabled
--kafka.producer.plaintext.username=user
--kafka.producer.plaintext.password=password

[foxylion]

We are seeing the same issue after updating to v1.66, v1.63 is working fine.

Our configuration is via env variables:

KAFKA_CONSUMER_BROKERS=....kafka.eu-central-1.amazonaws.com:9096,....kafka.eu-central-1.amazonaws.com:9096,....kafka.eu-central-1.amazonaws.com:9096
KAFKA_CONSUMER_TOPIC=jaeger-spans
KAFKA_CONSUMER_PLAINTEXT_USERNAME=jaeger
KAFKA_CONSUMER_AUTHENTICATION=none
KAFKA_CONSUMER_PLAINTEXT_PASSWORD=...

[amilbcahat]

hey @foxylion , @edefaria , I am taking this up , can you guys share me the exact command and setup you are trying for, it will help me out . Thanks !