Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Concurrency level limits performance #26

Open
pz07 opened this issue Mar 11, 2022 · 1 comment
Open

Concurrency level limits performance #26

pz07 opened this issue Mar 11, 2022 · 1 comment

Comments

@pz07
Copy link

pz07 commented Mar 11, 2022

Hi,
I've noticed the output is single-threaded. This limits performance significantly. In our case lag of sending log messages to DD is sometimes greater than 30 mins.
This is mainly caused by the fact that only one worker thread sends messages to DD. All other worker threads that finished processing inputs and filters are blocked.
This is due to the default concurrency level used by the plugin, which is :legacy.

Is it possible to change concurrency level of the plug-in to :shared to allow sending requests to DD in parallel?
The code change is described here: https://www.elastic.co/guide/en/logstash/current/output-new-plugin.html
From the doc:

# This sets the concurrency behavior of this plugin. By default it is :legacy, which was the standard
  # way concurrency worked before Logstash 2.4
  #
  # You should explicitly set it to either :single or :shared as :legacy will be removed in Logstash 6.0
  #
  # When configured as :single a single instance of the Output will be shared among the
  # pipeline worker threads. Access to the `#multi_receive/#multi_receive_encoded/#receive` method will be synchronized
  # i.e. only one thread will be active at a time making threadsafety much simpler.
  #
  # You can set this to :shared if your output is threadsafe. This will maximize
  # concurrency but you will need to make appropriate uses of mutexes in `#multi_receive/#receive`.
  #
  # Only the `#multi_receive/#multi_receive_encoded` methods need to actually be threadsafe, the other methods
  # will only be executed in a single thread
  concurrency :single

If you agree this is a good idea, I can do the change and create pull request.

I'm using Logstash v.6.x.x

@pz07
Copy link
Author

pz07 commented Mar 11, 2022

HTTP client the plug-in uses seems to be thread safe, so this seems to be simple configuration change
https://github.com/cheald/manticore

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants