Default buffer type to 'heap' for 9.0

[andsel]

Release notes

Switch the default value of pipeline.buffer.type to use the heap memory instead of direct one.

What does this PR do?

Change the default value of the setting pipeline.buffer.type from direct to heap and update consequently the documentation.

Why is it important/What is the impact to the user?

It's part of the work to make more explicit the usage of memory used by Netty based plugins. Using heap instead of direct for Netty buffers, make easier the debug of memory issues at the cost of bigger heap size.

Checklist

• My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Relates Set pipeline.buffer.type to heap by default #16353

Use cases

As a developer of Netty based plugin I want to have better insight in memory usage patterns.

[jsvd]

It is important to note that the Java heap sizing requirements will be impacted by this change since
allocations that previously resided on the direct memory will use heap instead. 
Performance-wise there shouldn't be a noticeable impact, since while direct memory IO is faster, Logstash Event objects produced by these plugins end up being allocated on the Java Heap, incurring the cost of copying from direct memory to heap memory regardless of the setting.

These must be changed as well, same with:

* When you set `pipeline.buffer.type` to `heap`, consider incrementing the Java heap by the 
amount of memory that had been reserved for direct space. 

[jsvd]

Please review the rest of the docs on the config-details asciidoc as it assumes direct memory is the default, that heap will be come the default, and that the user will be changing it to heap (which doesn't make sense for someone starting fresh with 9.x)

[yaauie]

Don't we use pooled buffers anyway? I'm under the impression that allocations of buffers may be faster in direct, but that our pool usage is a primary driver for our buffer use being performant. We also know that large buffer allocations can succeed in heap where they would fail in direct because the GC will rearrange the existing object space to ensure that it can allocate while the direct allocation will simply not have an appropriately-sized continuous chunk. I would rather a small performance cost in this case than a crashing OOM :)

Suggested change

	Performance-wise there shouldn't be a noticeable impact in switching to `direct`, since while direct memory IO is faster,
	Logstash Event objects produced by these plugins end up being allocated on the Java Heap,
	incurring the cost of copying from direct memory to heap memory regardless of the setting.
	Performance-wise there shouldn't be a noticeable impact in switching to `direct`.
	While allocating direct memory for an individual buffer is faster, these plugins use buffer pools to reduce allocations, and heap-managed allocations are significantly safer.
	Logstash Event objects produced by these plugins also end up being allocated on the Java Heap, incurring the cost of copying from buffers to heap memory regardless of the setting.

[andsel]

We always use the Netty PooledAllocator, but instead of using direct memory buffers it uses Java heap byte[]

[andsel]

Hi @yaauie

While allocating direct memory for an individual buffer is faster,

The is not the allocation that's faster, it's the transfer from OS buffers to direct buffers that's faster, that's usually needed when data is moved around in OS, so from network to filesystem, or network to network. In our case that direct buffers content always flow into Java heap space to inflate Logstash's events, so we loose the benefit of direct buffer, or at least is not so dominant.

[jsvd]

Minor tweaks, otherwise LGTM, but I'd like @karenzone's input here as well.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: d64d834

History

• 💚 Build #1701 succeeded a9098a4
• 💛 Build #1672 was flaky 44fdfaf
• 💛 Build #1650 was flaky d851770
• 💚 Build #1647 succeeded 6f4878e
• 💚 Build #1619 succeeded 54d1315
• 💛 Build #1615 was flaky be295b3

cc @andsel

[karenzone]

Left some suggestions for your consideration. Please LMKWYT.

[karenzone]

Suggested change

	Input plugins such as {agent}, {beats}, TCP, and HTTP will allocate buffers in Java heap memory to read events from the network.
	Input plugins such as {agent}, {beats}, TCP, and HTTP allocate buffers in Java heap memory to read events from the network.

[karenzone]

Use present tense instead of future tense when possible.

[karenzone]

Suggested change

	This is the preferred allocation method as it facilitates debugging memory usage problems (such as leaks and Out of Memory errors) through the analysis of heap dumps.
	Heap memory is the preferred allocation method, as it facilitates debugging memory usage problems (such as leaks and Out of Memory errors) through the analysis of heap dumps.

[karenzone]

For clarity and better SEO performance

[karenzone]

Suggested change

	Before version 9.0.0, Logstash defaulted to allocate direct memory for this purpose instead of heap. To re-enable the previous behaviour {ls} provides
	Before version 9.0.0, {ls} defaulted to direct memory instead of heap for this purpose. To re-enable the previous behavior {ls} provides

[karenzone]

I rearrange the words with the intention of simplifying the sentence. Please make sure that I didn't change the meaning.
Also, Elastic standard is US spelling.

[karenzone]

Suggested change

	There shouldn't be a noticeable performance impact in switching between `direct` and `heap`.
	Performance should not be noticeably affected if you switch between `direct` and `heap`.

[karenzone]

Suggested change

	While copying bytes from OS buffers to direct memory buffers is faster, Logstash Event objects produced by these plugins end up
	being allocated on the Java Heap incurring the cost of copying from direct memory to heap memory, regardless of the setting.
	While copying bytes from OS buffers to direct memory buffers is faster, {ls} Event objects produced by these plugins are allocated on the Java Heap, incurring the cost of copying from direct memory to heap memory, regardless of the setting.

[karenzone]

Suggested change

	# Defaults to heap, optionally can be switched to direct to select direct memory space.
	# Defaults to heap,but can be switched to direct if you prefer using direct memory space instead.