Kafka Producer Failing to Send Data – Network getting resetted

[VishnuLearnWeb]

Kafka Producer Failing to Send Data – Network Resets During Handshake

Hi Kafka Community,
I'm encountering a strange issue where a Kafka producer in my application is failing to send data to remote Kafka brokers.
However, other methods of sending data (like Kafka CLI tools and a custom client-server program) work fine.

Have 2 Scenarios:

Scenario 1 (Good Case) – Works Fine

• Machines are in the same remote location as the Kafka brokers.
• Data is sent to Kafka brokers without any interruptions.

Scenario 2 (Bad Case) – Issue Occurs

• Machines are in a different remote location, but Kafka brokers are reachable(pingable, SSH accessible).
• Using Kafka CLI tools (kafka-avro-console-producer and kafka-avro-console-consumer), data reaches Kafka without issues.
• A custom client-server program successfully sends data to the same Kafka broker ports.
• However, when using our application (which integrates Kafka producer libraries, messages fail to send, and we see connection failures.

Observed Errors (Kafka Producer Logs)

%5|1738751993.048|FAIL|rdkafka#producer-1| [thrd:sparc06:63000/bootstrap]: sparc06:63000/bootstrap: Connection closed
%3|1738751993.048|ERROR|rdkafka#producer-1| [thrd:sparc06:63000/bootstrap]: sparc06:63000/bootstrap: Connection closed
%3|1738751994.483|FAIL|rdkafka#producer-1| [thrd:sparc03:63000/bootstrap]: sparc03:63000/bootstrap: Receive failed: Connection reset by peer
%3|1738751994.483|ERROR|rdkafka#producer-1| [thrd:sparc03:63000/bootstrap]: sparc03:63000/bootstrap: Receive failed: Connection reset by peer
%3|1738751994.700|ERROR|rdkafka#producer-1| [thrd:sparc06:63000/bootstrap]: 3/3 brokers are down

Key Observations

• TCP dump shows that the network is getting reset during the handshake process
• In the server.properties, we have not enabled any security protocols.

advertised.listeners=PLAINTEXT://sparc03.ifelab.taus.us.thales:63000
# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

• Flushed the iptables, even then the issue persists.

• netstat output indicate that some connections are in SYN_SENT or ESTABLISHED, but they later drop/reset.
  Output of netstat:
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 1 10.113.50.109:51701 10.76.187.57:63000 SYN_SENT 0 596668609 5609/python3.6
  tcp 0 54 10.113.50.109:47559 10.76.187.62:63000 ESTABLISHED 0 596666333 5609/python3.6
  tcp 0 1 10.113.50.109:59312 10.76.187.59:63000 SYN_SENT 0 596668610 5609/python3.6
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 54 10.113.50.109:47571 10.76.187.62:63000 ESTABLISHED 0 596668635 5609/python3.6
  tcp 0 1 10.113.50.109:59324 10.76.187.59:63000 SYN_SENT 0 596668639 5609/python3.6
  tcp 0 0 10.113.50.109:51713 10.76.187.57:63000 ESTABLISHED 0 596667759 5609/python3.6
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 1 10.113.50.109:59331 10.76.187.59:63000 SYN_SENT 0 596667771 5609/python3.6
  tcp 0 1 10.113.50.109:51719 10.76.187.57:63000 SYN_SENT 0 596678688 5609/python3.6
  tcp 0 54 10.113.50.109:47577 10.76.187.62:63000 ESTABLISHED 0 596678678 5609/python3.6
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 1 10.113.50.109:47584 10.76.187.62:63000 SYN_SENT 0 596667823 5609/python3.6
  tcp 0 54 10.113.50.109:59334 10.76.187.59:63000 ESTABLISHED 0 596668685 5609/python3.6
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 54 10.113.50.109:59346 10.76.187.59:63000 ESTABLISHED 0 596678346 5609/python3.6
  tcp 0 1 10.113.50.109:47596 10.76.187.62:63000 SYN_SENT 0 596668705 5609/python3.6
  [root@i8000proto arts]# netstat -nape | grep -i '63000'
  tcp 0 1 10.113.50.109:47602 10.76.187.62:63000 SYN_SENT 0 596667865 -
  tcp 0 0 10.113.50.109:59352 10.76.187.59:63000 ESTABLISHED 0 596668718 5609/python3.6
  tcp 0 54 10.113.50.109:51741 10.76.187.57:63000 ESTABLISHED 0 596678399 5609/python3.6

• The issue only happens with our Kafka producer code inside the application, while other tools and programs work fine.

Questions & Help Needed

1. Why does this issue only occur with our Kafka producer application only in scenario 2? Other tools (CLI & custom program) work fine.
2. Why is the connection getting reset frequently during handshake? Could this be related to Kafka’s session handling, retries, or network configuration?
3. Could firewall, NAT, or Kafka security settings (like SSL, SASL) be causing this behavior?
4. Any Kafka producer configuration tweaks (e.g., retries, linger.ms, connections.max.idle.ms) that might help?

Would appreciate any insights on this! Thanks in advance.