[8.x] [filebeat] Elasticsearch state storage for httpjson and cel inputs (#41446) #42468
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lastic#41446) This enables Elasticsearch as State Store Backend for Security Integrations for the Agentless solution. The scope of this change was narrowed down to supporting only `httpjson` inputs in order to support Okta integration for the initial release. All the other integrations inputs still use the file storage as before. This is a short term solution for the state storage for k8s. The feature currently can only be enabled with the `AGENTLESS_ELASTICSEARCH_STATE_STORE_INPUT_TYPES` env var. The existing code relied on the inputs state storage to be fully configurable before the main beat managers runs. The change delays the configuration of `httpjson` input to the time when the actual configuration is received from the Agent. Example of the state storage index content for Okta integration: ``` { "took": 6, "timed_out": false, "_shards": { "total": 1, "successful": 1, "skipped": 0, "failed": 0 }, "hits": { "total": { "value": 1, "relation": "eq" }, "max_score": 1, "hits": [ { "_index": "agentless-state-httpjson-okta.system-028ecf4b-babe-44c6-939e-9e3096af6959", "_id": "httpjson::httpjson-okta.system-028ecf4b-babe-44c6-939e-9e3096af6959::https://dev-36006609.okta.com/api/v1/logs", "_seq_no": 39, "_primary_term": 1, "_score": 1, "_source": { "v": { "ttl": 1800000000000, "updated": "2024-10-24T20:21:22.032Z", "cursor": { "published": "2024-10-24T20:19:53.542Z" } } } } ] } } ``` The naming convention for all state store is `agentless-state-<input id>`, since the expectation for agentless we would have only one agent per policy and the agents are ephemeral. Closes https://github.com/elastic/security-team/issues/11101 Co-authored-by: Orestis Floros <[email protected]> (cherry picked from commit 8180f23) # Conflicts: # filebeat/beater/filebeat.go
orestisfl
requested review from
mauri870 and
faec
and removed request for
a team
6 tasks
botelastic
bot
added
the
needs_team
|
❌ Author of the following commits did not sign a Contributor Agreement: Please, read and sign the above mentioned agreement if you want to contribute to this project |
|
This pull request doesn't have a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto8.x:Questions ?
Please refer to the Backport tool documentation