Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] [AWS] add support to source logs from AWS linked source accounts when using log_group_name_prefix #41206

Merged
merged 6 commits into from
Oct 15, 2024
Merged

[Filebeat] [AWS] add support to source logs from AWS linked source accounts when using log_group_name_prefix #41206

merged 6 commits into from
Oct 15, 2024

Conversation

Kavindu-Dodan
Copy link
Contributor

@Kavindu-Dodan Kavindu-Dodan commented Oct 11, 2024 •
edited
Loading

Proposed commit message

This is a follow-up to #41188 where I am adding support to source linked accounts when using log_group_name_prefix to derive log groups.

PR introduce include_linked_accounts_for_prefix_mode boolean property, which is disabled by default. If enabled (include_linked_accounts_for_prefix_mode : true), then we set includeLinkedAccounts property of the DescribeLogGroups API [1] to obtain log groups matching prefix and included in linked accounts of the monitoring account.

ex:-

- type: aws-cloudwatch
  ...
  log_group_name_prefix : /development/AppA/
  include_linked_accounts_for_prefix_mode: true
  ...

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

This require a linked cloudwatch account. If already has one, then,

  • Push logs to a newly created log group OR use an already existing log group in a source account
    • Note - you may use data-gen Go program to generate and push logs to your log group (using output CLOUDWATCH_LOG) [2]
  • Configure filebeat cloudwatch input with log_group_name_prefix with desired prefix & set include_linked_accounts_for_prefix_mode to value true (enabled)
  • Run filebeat and observe logs in Kibana discover which include logs from log groups (that match provided prefix)

Related issues

[1] - https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogGroups.html
[2] - https://github.com/Kavindu-Dodan/data-gen

@Kavindu-Dodan Kavindu-Dodan requested review from a team as code owners October 11, 2024 15:58
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 11, 2024
@Kavindu-Dodan Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Oct 11, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 11, 2024
@Kavindu-Dodan Kavindu-Dodan added needs_team Indicates that the issue/PR needs a Team:* label backport-8.x Automated backport to the 8.x branch with mergify labels Oct 11, 2024
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 11, 2024
@Kavindu-Dodan Kavindu-Dodan mentioned this pull request Oct 11, 2024
Merged
6 tasks
@kaiyan-sheng
Copy link
Contributor

Just one comment for now: what do you think about the name include_inked_accounts_for_prefix_mode? just to match include_linked_accounts in metricbeat.

@Kavindu-Dodan
Copy link
Contributor Author

Just one comment for now: what do you think about the name include_inked_accounts_for_prefix_mode? just to match include_linked_accounts in metricbeat.

Good suggestion, done with commit 3d3b46f

@pierrehilbert pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Oct 13, 2024
@elasticmachine
Copy link
Collaborator

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@pierrehilbert pierrehilbert requested review from rdner and faec and removed request for AndersonQ and leehinman October 13, 2024 12:40
@kaiyan-sheng kaiyan-sheng mentioned this pull request Oct 14, 2024
Closed
@Kavindu-Dodan
configuration parsing to support arn & linked accounts
f749273 
Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/input.go
@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/use-linked-account-when-using-log-group-prefix branch from 3d3b46f to 2d0b247 Compare October 15, 2024 14:26
@Kavindu-Dodan
add changelog entry
4049180 
Signed-off-by: Kavindu Dodanduwa <[email protected]>
@Kavindu-Dodan
review suggestion
4cd801e 
Signed-off-by: Kavindu Dodanduwa <[email protected]>
@Kavindu-Dodan Kavindu-Dodan force-pushed the feat/use-linked-account-when-using-log-group-prefix branch from 2d0b247 to 4cd801e Compare October 15, 2024 14:30
@Kavindu-Dodan
use non-pointer struct property
7d37eb8 
Signed-off-by: Kavindu Dodanduwa <[email protected]>
@elastic elastic deleted a comment from mergify bot Oct 15, 2024
@Kavindu-Dodan Kavindu-Dodan merged commit 7e1b528 into elastic:main Oct 15, 2024
22 checks passed
mergify bot pushed a commit that referenced this pull request Oct 15, 2024
@Kavindu-Dodan @mergify
[Filebeat] [AWS] add support to source logs from AWS linked source ac…
fc8792a 
…counts when using log_group_name_prefix (#41206)

* configuration parsing to support arn & linked accounts

Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/input.go

* code review change - fix typo

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add support to linked accounts when using prefix mode

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add changelog entry

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* review suggestion

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* use non-pointer struct property

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
(cherry picked from commit 7e1b528)
@mergify mergify bot mentioned this pull request Oct 15, 2024
Merged
6 tasks
pierrehilbert pushed a commit that referenced this pull request Oct 15, 2024
@mergify @Kavindu-Dodan
[Filebeat] [AWS] add support to source logs from AWS linked source ac…
ac439c8 
…counts when using log_group_name_prefix (#41206) (#41247)

* configuration parsing to support arn & linked accounts

Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/input.go

* code review change - fix typo

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add support to linked accounts when using prefix mode

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add changelog entry

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* review suggestion

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* use non-pointer struct property

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
(cherry picked from commit 7e1b528)

Co-authored-by: Kavindu Dodanduwa <[email protected]>
@Kavindu-Dodan Kavindu-Dodan mentioned this pull request Oct 17, 2024
Open
belimawr pushed a commit to belimawr/beats that referenced this pull request Oct 18, 2024
@Kavindu-Dodan @belimawr
[Filebeat] [AWS] add support to source logs from AWS linked source ac…
9b6f2e1 
…counts when using log_group_name_prefix (elastic#41206)

* configuration parsing to support arn & linked accounts

Signed-off-by: Kavindu Dodanduwa <[email protected]>

# Conflicts:
#	x-pack/filebeat/input/awscloudwatch/input.go

* code review change - fix typo

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add support to linked accounts when using prefix mode

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* add changelog entry

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* review suggestion

Signed-off-by: Kavindu Dodanduwa <[email protected]>

* use non-pointer struct property

Signed-off-by: Kavindu Dodanduwa <[email protected]>

---------

Signed-off-by: Kavindu Dodanduwa <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@belimawr belimawr belimawr approved these changes

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

@rdner rdner Awaiting requested review from rdner

@faec faec Awaiting requested review from faec

Assignees

@Kavindu-Dodan Kavindu-Dodan

Labels
backport-8.x Automated backport to the 8.x branch with mergify Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team Team:obs-ds-hosted-services Label for the Observability Hosted Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Filebeat] [AWS] Support getting cloudwatch logs from linked cross-account monitoring source accounts
5 participants
@Kavindu-Dodan @elasticmachine @kaiyan-sheng @belimawr @pierrehilbert