[auditbeat] fim: implement kprobes backend #37796

pkoutsovasilis · 2024-01-31T03:04:54Z

Proposed commit message

This PR adds an additional opt-in kprobes (through tracefs) backend to the auditbeat's File Integrity Module (FIM). This is the first part in a PR series that just introduces this new FIM backend and captures only the pid of the process that caused the respective file change. As soon as this one gets merged to main, another one will follow as an extension to this one that captures also the uid and gid of the process. This FIM BE serves as a fallback for older linux kernels (3.10-5.10.15) where the eBPF FIM BE (#37223) is not supported.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

open-source https://github.com/elastic/tk-btf

How to test this PR locally

cd auditbeat && mage build integTest

TEST-go-integration.out.json (kprobe package coverage 85%)
TEST-python-integration.xml.txt

One observation is that mage build integTest is running all unit-tests and python integration tests inside containers using the docker-compose.yml of auditbeat package, thus everything is tested. However, in the CI pipelines mage build unitTest is invoked which performs the same steps, unit-tests, python integration test but as this is invoked with no root the kprobe fim unit-tests and integration tests can't be invoked.

Related issues

File Integrity Monitoring | User Information - Linux integrations#7401

…s thread

…f ones

…tr tk-btf probe builders in probe manager

… path

…directory entry cache emit the respective event

…nce of generated fs events are properly emitted

…and satisfy testing needs

…el and event emitting

…ew monitor based on these

…ting it

mergify · 2024-01-31T03:05:31Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @pkoutsovasilis? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2024-01-31T03:05:39Z

Pinging @elastic/sec-linux-platform (Team:Security-Linux Platform)

…er-compose.yml

auditbeat/module/file_integrity/kprobes/path_inotify_test.go

auditbeat/module/file_integrity/kprobes/path_mountpoint.go

auditbeat/module/file_integrity/kprobes/path_test.go

auditbeat/module/file_integrity/kprobes/verifier.go

mergify · 2024-02-12T23:06:53Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b pkoutsovasilis/kprobe_fim upstream/pkoutsovasilis/kprobe_fim
git merge upstream/main
git push upstream pkoutsovasilis/kprobe_fim

andrewkroh

In a follow up PR we should implement some metrics to improve observability and troubleshooting. The BaseMetricSet should provide a metric namespace for the input to hook into for adding new metrics that get exposed on the /inputs/ monitoring API.

Would you mind gofumpting this new package please.

I did quick test on Ubuntu 22.04 (Linux linux 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:26:57 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux) with the kprobes backend enabled. It appears to be working as expected. 👍 Looking forward to the addition of user info next.

andrewkroh · 2024-02-13T19:24:57Z

auditbeat/docker-compose.yml

@@ -14,6 +14,7 @@ services:
      - KIBANA_PORT=5601
    volumes:
      - ${PWD}/..:/go/src/github.com/elastic/beats/
+      - /sys:/sys


Auditbeat has doc page about running in Docker. For a future PR, I think that page should get updated to specify the specific settings (capabilities, user, mounts, etc) that are necessary for each FIM backend.

oops noted, I wasn't aware of that but definitely more than a valid think to look out, especially when kprobe BE of FIM transitions out from technical preview, thx for the catch!

auditbeat/module/file_integrity/kprobes/events_process.go

# Conflicts: # auditbeat/auditbeat.reference.yml # auditbeat/docs/modules/file_integrity.asciidoc # auditbeat/module/file_integrity/_meta/config.yml.tmpl # auditbeat/module/file_integrity/_meta/docs.asciidoc # auditbeat/module/file_integrity/config.go # auditbeat/module/file_integrity/event.go # auditbeat/module/file_integrity/eventreader_fsnotify.go # auditbeat/module/file_integrity/eventreader_linux.go # auditbeat/module/file_integrity/eventreader_other.go # auditbeat/module/file_integrity/schema.fbs # auditbeat/module/file_integrity/schema/Source.go # auditbeat/tests/system/test_file_integrity.py # go.mod # x-pack/auditbeat/auditbeat.reference.yml

…time to verify the probes and print output

pkoutsovasilis · 2024-02-13T23:21:33Z

In a follow up PR we should implement some metrics to improve observability and troubleshooting. The BaseMetricSet should provide a metric namespace for the input to hook into for adding new metrics that get exposed on the /inputs/ monitoring API.

Oh thx, metrics is such a nice idea, definitely we will implement them!

Would you mind gofumpting this new package please.

gofumpted

I did quick test on Ubuntu 22.04 (Linux linux 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9 15:26:57 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux) with the kprobes backend enabled. It appears to be working as expected. 👍 Looking forward to the addition of user info next.

you are officially the first tech preview tester 😁

elasticmachine · 2024-02-13T23:34:21Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💚 Build #1351 succeeded 6cd08cb
💚 Build #1207 succeeded da84277
💚 Build #1206 succeeded fe6453a

cc @pkoutsovasilis

elasticmachine · 2024-02-13T23:45:07Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💚 Build #1344 succeeded 6cd08cb
💚 Build #1202 succeeded da84277
💔 Build #1201 failed fe6453a

cc @pkoutsovasilis

elasticmachine · 2024-02-13T23:54:03Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💚 Build #515 succeeded 6cd08cb
💚 Build #371 succeeded da84277

cc @pkoutsovasilis

elasticmachine · 2024-02-14T00:01:49Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💚 Build #2420 succeeded da84277

cc @pkoutsovasilis

elasticmachine · 2024-02-14T00:05:39Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💚 Build #1532 succeeded da84277
💔 Build #1531 failed fe6453a

cc @pkoutsovasilis

elasticmachine · 2024-02-14T00:18:07Z

💚 Build Succeeded

Buildkite Build
Commit: f1e51f4

History

💔 Build #3010 failed bd8d23a
💚 Build #2871 succeeded da84277
💔 Build #2411 failed 3ad318b

cc @pkoutsovasilis

pkoutsovasilis and others added 17 commits January 25, 2024 17:14

feat: add helper funcs to get symbol info from /proc/kallsyms

cb01f59

feat: introduce fixed executor that always runs funcs from the same o…

9417243

…s thread

feat: add probe manager to handle building tracing kprobes from tk-bt…

bbca9e7

…f ones

feat: define probe events with corresponding alloc and release funcs

292b7b7

feat: embed stripped btf files and add helper funcs to read them

d585da1

feat: add fsnotify, fsnotify_nameremove, fsnotify_parent and vfs_geat…

e4616c1

…tr tk-btf probe builders in probe manager

feat: implement path traverser to produce monitor events by walking a…

625da6b

… path

feat: implement directory entries cache

6f35ab1

feat: implement event processor to process probe events and based on …

31ec585

…directory entry cache emit the respective event

feat: implement event verifier that validates that the expected seque…

ea61593

…nce of generated fs events are properly emitted

feat: add perfChannel to reduce tracing.PerfChannel boilerplate code …

f58e369

…and satisfy testing needs

feat: implement monitor that ties together path traverser, perf chann…

0c785ca

…el and event emitting

feat: implement probe verification at runtime and the creation of a n…

d8bf292

…ew monitor based on these

feat: implement event reader for kprobe-based file integrity module

097aa25

doc: update NOTICE.txt to include tk-btf license

c4a9d9b

feat: add tests for non-recursive kprobe fim (#3)

6ca359f

fix: remove existing file from cache when a move operation is overwri…

07b927b

…ting it

pkoutsovasilis requested review from a team as code owners January 31, 2024 03:04

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jan 31, 2024

mergify bot assigned pkoutsovasilis Jan 31, 2024

pkoutsovasilis added the Team:Security-Linux Platform Linux Platform Team in Security Solution label Jan 31, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jan 31, 2024

feat: introduce force_backend in for file integrity auditbeat module

045bf40

pkoutsovasilis force-pushed the pkoutsovasilis/kprobe_fim branch 3 times, most recently from b4efcc5 to 46ef848 Compare January 31, 2024 18:17

ci: add necessary volume mounts for kprobes backend in auditbeat dock…

bd6bcfb

…er-compose.yml