[Filebeat] [AWS] Support getting cloudwatch logs from linked cross-account monitoring source accounts

[tommyers-elastic]

We should support cross-account log collection for Cloudwatch. We almost have this functionality today, but there is a small hurdle to overcome, as detailed below.

Cross-account logs are supported by the FilterLogEvents API (which we already use). However there is a caveat hidden away in the docs:

If the log group is in a source account and you are using a monitoring account, you must use the log group ARN.

If the log group name is used for log groups in linked source accounts, the request does not succeed. In our cloudwatch logs input, we utilize the log group name exclusively (even parsing the name from the ARN, if provided https://github.com/elastic/beats/blob/main/x-pack/filebeat/input/awscloudwatch/input.go#L72).

In order for cross-account monitoring to work for cloudwatch logs, we need to switch to using log group ARNs and
logGroupIdentifier in the API requests, instead of log group names (and logGroupName).

[hesingon]

Can this be addressed soon?

[kaiyan-sheng]

Hey @tommyers-elastic do you mind if we take over the PR and continue working on it?

[Kavindu-Dodan]

I started looking into this. Note that, while logGroupIdentifier allows us to accept an ARN, it has a regex pattern restriction of [\w#+=/:,.@-]* [1]. This patterns doesn't allow suffix * (wildcard) that's present in for example aws logs describe-log-groups or cloudwatch UI.

However, once wildcard is removed, logGroupIdentifier works as expected.

[1] - https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_FilterLogEvents.html

[Kavindu-Dodan]

Consider this thread [1]. DescribeLogGroups API [2] allows to configure includeLinkedAccounts property which is disabled by default. This means, when users use logGroupNamePrefix, we can still support including logs from linked accounts.

However, we must control this behaviour through a configuration so that existing users do not get impacted with extra logs. Hence, the property should be disabled by default. And should work in-combination with logGroupNamePrefix

Proposed property name - linked_accounts_for_prefix_mode

[1] - #41188 (comment)
[2] - https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_DescribeLogGroups.html

[nsshah1288]

Thank you so much @Kavindu-Dodan @kaiyan-sheng and everyone else for working on this. Today, I was able to use the elastic agent to pull in logs from a separate AWS account!! For FYI in case it helps anyone else, I am not using CloudWatch cross-account observability. I exec'd into one of the elastic agent pods and created a credentials file located at ~/.aws/credentials, which looked like this:

[default]
role_arn = arn:aws:iam::<core-account-id>:role/elastic-agent-role
web_identity_token_file = /var/run/secrets/eks.amazonaws.com/serviceaccount/token
[profile my-profile]
source_profile = default
role_arn = arn:aws:iam::<target-account-id>:role/elastic-agent-role

where core-account-id is the central account where elastic is deployed and target-account-id is the account where there are CloudWatch logs I want to pull in.

I used IRSA and followed this AWS blog: https://aws.amazon.com/blogs/containers/enabling-cross-account-access-to-amazon-eks-cluster-resources/ , which ensured that the IAM roles have the necessary permissions and trust relationships.

Then, in the AWS CloudWatch integration page in Kibana, I told it to use my-profile, and the arn:aws:iam::<target-account-id>:role/elastic-agent-role as the Role ARN, and then the logs from target-account-id were pulled in to elastic!