x-pack/filebeat/input/cel: add ability to remove request trace logs #39969

efd6 · 2024-06-20T01:14:49Z

Proposed commit message

The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed.

The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

Similar changes should be made in other inputs that write trace logs.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

[ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2024-06-20T03:28:19Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

chrisberkhout

Cleaning up sensitive data seems like a good idea, but on the other hand I think there's a good argument to be made that comprehensive management of the log files it generates is beyond the scope of beats, especially since unlinking the file will probably leave the data on the block device, and attempting to overwrite it would probably still keep it around (due to CoW file systems or SSD leveling).

There are some issues to work out with the implementation.

x-pack/filebeat/input/cel/input_test.go

x-pack/filebeat/input/cel/input.go

chrisberkhout · 2024-06-20T16:59:41Z

Forgot to mention I do like the choice of a new option and how it's done in a backwards-compatible way.

mergify · 2024-06-20T21:13:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b remove_trace_logs_cel upstream/remove_trace_logs_cel
git merge upstream/main
git push upstream remove_trace_logs_cel

chrisberkhout

Looks good!

mergify · 2024-06-21T21:08:42Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b remove_trace_logs_cel upstream/remove_trace_logs_cel
git merge upstream/main
git push upstream remove_trace_logs_cel

The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

This is a replay of elastic#39969, but for the HTTPJSON input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs This is essentially a replay of elastic#39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ace logs This is essentially a replay of elastic#39969, but for the http_endpoint input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ogs (#40003) This is a replay of #39969, but for the HTTPJSON input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ace logs This is essentially a replay of elastic#39969, but for the http_endpoint input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs This is essentially a replay of elastic#39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ace logs This is essentially a replay of elastic#39969, but for the http_endpoint input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs This is essentially a replay of elastic#39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ace logs This is essentially a replay of elastic#39969, but for the http_endpoint input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs This is essentially a replay of elastic#39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…ace logs (#40005) This is essentially a replay of #39969, but for the http_endpoint input. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs This is essentially a replay of elastic#39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

…dd ability to remove request trace logs (#40004) This is essentially a replay of #39969, but for the entity analytics providers. The previous configuration system did not allow users to remove trace logs from agents after they are no longer needed. This is potential security risk as it leaves potentially sensitive information on the file system beyond its required lifetime. The mechanism for communicating to the input whether to write logs is not currently powerful enough to indicate that existing logs should be removed without deleting logs from other instances. So add an enabled configuration option to allow the target name to be sent independently of whether the files should be written or removed. The new option is optional, defaulting to the previous behaviour so that it can be opted into via progressive repair in the client integrations.

efd6 added enhancement Filebeat Filebeat backport-skip Skip notification from the automated backport with mergify Team:Security-Service Integrations Security Service Integrations Team labels Jun 20, 2024

efd6 self-assigned this Jun 20, 2024

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jun 20, 2024

efd6 force-pushed the remove_trace_logs_cel branch 2 times, most recently from b226c1e to 8952f2c Compare June 20, 2024 01:16

efd6 marked this pull request as ready for review June 20, 2024 03:28

efd6 requested a review from a team as a code owner June 20, 2024 03:28

chrisberkhout requested changes Jun 20, 2024

View reviewed changes

x-pack/filebeat/input/cel/input_test.go Show resolved Hide resolved

x-pack/filebeat/input/cel/input.go Show resolved Hide resolved

x-pack/filebeat/input/cel/input.go Outdated Show resolved Hide resolved

efd6 force-pushed the remove_trace_logs_cel branch from 4cf563c to 405ea2e Compare June 20, 2024 21:46

chrisberkhout approved these changes Jun 21, 2024

View reviewed changes

efd6 added 2 commits June 22, 2024 06:40

address pr comments

e6bfa40

efd6 force-pushed the remove_trace_logs_cel branch from 405ea2e to e6bfa40 Compare June 21, 2024 21:10

efd6 enabled auto-merge (squash) June 21, 2024 21:12

efd6 merged commit 8b89f79 into elastic:main Jun 21, 2024
15 of 18 checks passed

efd6 mentioned this pull request Jun 25, 2024

x-pack/filebeat/input/httpjson: add ability to remove request trace logs #40003

Merged

6 tasks

efd6 mentioned this pull request Jun 25, 2024

x-pack/filebeat/input/entityanalytics/{okta,azuread/fetcher/graph}: add ability to remove request trace logs #40004

Merged

6 tasks

efd6 mentioned this pull request Jun 25, 2024

x-pack/filebeat/input/http_endpoint: add ability to remove request trace logs #40005

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

x-pack/filebeat/input/cel: add ability to remove request trace logs #39969

x-pack/filebeat/input/cel: add ability to remove request trace logs #39969

efd6 commented Jun 20, 2024 •

edited

Loading

elasticmachine commented Jun 20, 2024

chrisberkhout left a comment

chrisberkhout commented Jun 20, 2024

mergify bot commented Jun 20, 2024

chrisberkhout left a comment

mergify bot commented Jun 21, 2024

x-pack/filebeat/input/cel: add ability to remove request trace logs #39969

x-pack/filebeat/input/cel: add ability to remove request trace logs #39969

Conversation

efd6 commented Jun 20, 2024 • edited Loading

Proposed commit message

Checklist

Disruptive User Impact

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Jun 20, 2024

chrisberkhout left a comment

Choose a reason for hiding this comment

chrisberkhout commented Jun 20, 2024

mergify bot commented Jun 20, 2024

chrisberkhout left a comment

Choose a reason for hiding this comment

mergify bot commented Jun 21, 2024

efd6 commented Jun 20, 2024 •

edited

Loading