Logstash does not progress and requires reloading for progressing with data

[anmironov]

logstash-input-tcp (6.3.1)
Using bundled JDK: /usr/share/logstash/jdk

[2024-05-15T23:06:04,588][ERROR][logstash.javapipeline    ][name][d1f632ab5f37317b4d81a4bd86c8b71d69714f60f6cdc15ff9d0091a1d86768f] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:name
  Plugin: <LogStash::Inputs::Tcp mode=>"client", host=>"X.X.X.X", codec=><LogStash::Codecs::Multiline pattern=>"\\<---\\>", what=>"next", id=>"13b4160f-2a02-472e-91f7-cc28c5e78ed9", max_lines=>20, negate=>true, enable_metric=>true, charset=>"UTF-8", multiline_tag=>"multiline", max_bytes=>10485760>, id=>"d1f632ab5f37317b4d81a4bd86c8b71d69714f60f6cdc15ff9d0091a1d86768f", port=>8765, enable_metric=>true, proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>, tcp_keep_alive=>false, dns_reverse_lookup_enabled=>true>
  Error: Connection refused - connect(2) for "10.192.222.6" port 8765
  Exception: Errno::ECONNREFUSED
  Stack: org/jruby/ext/socket/RubyTCPSocket.java:134:in `initialize'
org/jruby/RubyClass.java:911:in `new'
org/jruby/RubyIO.java:868:in `new'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-tcp-6.3.1-java/lib/logstash/inputs/tcp.rb:349:in `new_client_socket'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-tcp-6.3.1-java/lib/logstash/inputs/tcp.rb:217:in `run_client'
/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-input-tcp-6.3.1-java/lib/logstash/inputs/tcp.rb:169:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:411:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:402:in `block in start_input'

LS restart is a workaround

Possible pain point: tcp-input is missing retry in client mode. When the initial connection fails, it should retry.

[martinotisatec]

Good afternoon,
This is issue is critically afecting the production service.
Kindly, speed up the resolution process.

Sincerely,

[kaisecheng]

@martinotisatec After testing in 7.17 and 8.15, I can see the plugin has restarted, retried and successfully reconnected to server when Errno::ECONNREFUSED.

We need reproducible steps to investigate the issue. Can you provide a smallest possible pipeline and steps to reproduce the problem?

[martinotisatec]

Good afternoon @kaisecheng,
Please provide the steps to obtain a useful pipeline.
Does it require to run into this issue?

Sincerely,

[kaisecheng]

@martinotisatec This can significantly speed up the investigation and apply the fix, otherwise, I would suggest to create a support case to have a deeper analyst.