Add new agentbeat with all beats shipped with Elastic Agent as a single beat

[blakerouse]

Proposed commit message

Add a new agentbeat that compiles into a single binary that provides subcommands for each contained beat. Contains auditbeat, filebeat, heartbeat, metricbeat, osquerybeat, and packetbeat.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc. being this is just a combined beat, don't think we need to add a changelog for it (it just gets all the changes the other beats get)

Author's Checklist

• auditbeat works
• filebeat works
• heartbeat works
• metricbeat works
• osquerybeat works
• packetbeat works

How to test this PR locally

$ cd x-pack/agentbeat
$ mage package
$ cd build/distribution
$ tar zxvf agentbeat*.tar.gz
$ cd agentbeat*
$ ./agentbeat filebeat run -e

Related issues

Use cases

Provides a much smaller binary when all of the beats are combined into a single binary instead of each in there own binary.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @blakerouse? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-04-11T19:03:00.568+0000

• Duration: 362 min 25 sec

Test stats 🧪

Test	Results
Failed	13
Passed	21696
Skipped	1470
Total	23179

Test errors

Expand to view the tests failures

> Show only the first 10 test failures

Build&Test / auditbeat-rhel-9-rhel-9 / TestImmutable – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestImmutable
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestImmutable (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestData – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestData
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestData (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestLoginType – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestLoginType
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestLoginType (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/auditlogin – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/auditlogin
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/auditlogin (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/centos7 – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/centos7
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/centos7 (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/chown – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/chown
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/chown (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/passwd – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/passwd
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/passwd (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/setuid – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/setuid
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/setuid (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/sudo-asuser – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/sudo-asuser
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/sudo-asuser (0.00s)
 

Build&Test / auditbeat-rhel-9-rhel-9 / TestGoldenFiles/sudo – github.com/elastic/beats/v7/auditbeat/module/auditd

Expand to view the error details

 Failed 

Expand to view the stacktrace

 === RUN   TestGoldenFiles/sudo
    modules.go:126: failed to create new MetricSet no metricsets configured for module 'auditd'
--- FAIL: TestGoldenFiles/sudo (0.00s)
 

Steps errors

Expand to view the steps failures

Show only the first 10 steps failures

auditbeat-rhel-9-rhel-9 - mage build unitTest

• Took 1 min 43 sec . View more details here
• Description: mage build unitTest

auditbeat-rhel-9-rhel-9 - mage build unitTest

• Took 1 min 43 sec . View more details here
• Description: mage build unitTest

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Checks if running on a Unix-like node

• Took 0 min 0 sec . View more details here

Error signal

• Took 0 min 0 sec . View more details here
• Description: Error 'hudson.AbortException: script returned exit code 1'

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[belimawr]

That looks great @blakerouse !
Did you measure the disk savings? It would be nice to have it in the PR/Commit.

[leehinman]

Do any of the fields change when you ingest events with agentbeat? I'm noticing some name changes in monitoring, input, and module names and I'm curious if that changes the emitted events in any way.

[blakerouse]

That looks great @blakerouse ! Did you measure the disk savings? It would be nice to have it in the PR/Commit.

700M - combined total for auditbeat, filebeat, heartbeat, metricbeat, osquerybeat, packetbeat
207M - agentbeat (single build with auditbeat, filebeat, heartbeat, metricbeat, osquerybeat, packetbeat all in one)

493M in savings a reduction of 70% for just those beats.

Note: This was done on darwin/arm64. Each OS/arch will have a different result, but it should be in proportion.

[blakerouse]

Do any of the fields change when you ingest events with agentbeat? I'm noticing some name changes in monitoring, input, and module names and I'm curious if that changes the emitted events in any way.

@leehinman That is a great question, I had to adjust the auditbeat system module name to audit/system because it collides with the same name of system with metricbeat. It only changes the registered name, I would hope that it doesn't affect the actual events being published. I don't know, I am no expert in how metricbeat works and if the registered module name affects the published events.

There is also an adjustment in the internal metrics reported for the beats, but its only for the name of the key for the running modules, and inputs. I don't believe this will have any affect, other than for debugging.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 777e720

History

• 💚 Build #963 succeeded c17068c
• 💚 Build #942 succeeded ee5310c
• 💚 Build #940 succeeded 19eead4
• 💚 Build #939 succeeded 3cc5471

cc @blakerouse

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 777e720

History

• 💚 Build #1800 succeeded c17068c
• 💚 Build #1779 succeeded ee5310c
• 💚 Build #1777 succeeded 19eead4
• 💚 Build #1776 succeeded 3cc5471

cc @blakerouse

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 777e720

Failed CI Steps

• :windows:-family/core-windows-2016 Unit Tests

History

• 💔 Build #3436 failed c17068c
• 💚 Build #3415 succeeded ee5310c

cc @blakerouse

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 777e720

History

• 💚 Build #2075 succeeded ee5310c

cc @blakerouse

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 777e720

History

• 💚 Build #1786 succeeded ee5310c

cc @blakerouse

[pazone]

Are we going to merge it soon? Or can use this branch to create the pipeline?

[blakerouse]

@andrewkroh I was able to adjust agentbeat and each of the beats to only register what each need. You will see the addition of InitializeModule and the automated way it is handled by mage to generate the code to call the initialization. This logic is only applied to areas where its critical that the original func init() is called when that specific beat is being used.

This change ensures that processors will not cross other beats, adjustments to kuberentes/docker processors will only take affect for each beat.

Another change is that auditbeat now uses its own registry to ensure that it doesn't collide with metricbeat. This allowed removing the changes that adjusted the name of the module and metricsets to ensure that when agentbeat auditbeat is called for the system module it used the system module for auditbeat, and when its agentbeat metricbeat its the system module for metricbeat.

[blakerouse]

@pazone I am waiting on a green CI and then I will get this merged so the PR for building can be created.

[cmacknz]

So we can't use init() anymore without side effects for agentbeat?

I don't mind it, but this seems like something that will be forgotten or undone without some kind of automated enforcement.

[blakerouse]

Correct, caution will need to be taken in the case that a beat is modifying a behavior that is coming from libbeat.

If the change can bleed into another beat then you need to use InitializeModule, otherwise init can be used.

Like init is still being used in most places, but only the areas where it can cause an effect was it changed to InitializeModule.

[blakerouse]

It is also really hard to enforce with automation, because its conditional if it is required. It all depends on the context of the change.

It applied to anything that is modifying a behavior that is coming from libbeat or x-pack/libbeat.

[blakerouse]

All tests pass that are run by Jenkins, this is ready to be merged if okay with the implementation and change.

@pazone Is blocked on getting this merged to create the packaging job.

[elasticmachine]

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

[elasticmachine]

Pinging @elastic/sec-deployment-and-devices (Team:Security-Deployment and Devices)

[andrewkroh]

Assume it exists, can someone please point me at the buildkite job that periodically runs all Fleet integrations' system tests using the latest elastic-agent snapshot. I want to keep any on that as soon as this become available in a 8.14.0-SNAPSHOT.

[cmacknz]

Might be this one https://buildkite.com/elastic/elastic-package-test-with-integrations.

[cmacknz]

Merging this PR on its own isn't enough to update the agent build so you won't see any impact yet.

[cmacknz]

Merging so we can get the build pipeline and additional validation started.

[andrewkroh]

Might be this one https://buildkite.com/elastic/elastic-package-test-with-integrations.

That pointed me in the right direction. I think it is this one. But we don't have any testing with the next snapshot.

https://buildkite.com/elastic/integrations-schedule-daily/builds/169

[blakerouse]

PR for packaging - #38880

[blakerouse]

If your looking for an early build artifact - https://buildkite.com/elastic/beats-xpack-agentbeat-package/builds/17#018ed2c8-b857-429e-9d5c-f38f264fcc14

[lucabelluccini]

So this binary will fulfil the capabilities to implement the inputs of integrations?
Do we have coverage on each integration will be started correctly?

[cmacknz]

So this binary will fulfil the capabilities to implement the inputs of integrations?

Yes, it is all of the Beats agent currently runs compiled into a single binary, with the individual beats accessible via sub-commands e.g agentbeat filebeat run -e should be equivalent to filebeat run -e

Do we have coverage on each integration will be started correctly?

I am confident we don't have 100% test coverage for every feature of every integration, however the scope of what can go wrong with this change should be limited to interactions between the global state of each Beat. There are not many things in this category, most of them have been raised in this PR already. There are always unknown unknowns of course.

[andrewkroh]

PR to begin system testing with 8.14.0-SNAPSHOT for all integrations on buildkite: elastic/integrations#9585