Add support fof 'sc2s.sgov.gov' in filebeats awss3 input

[mgerba]

What does this PR do?

Added the 'sc2s.sgov.gov' domain to Xpack Filebeats awss3 input.go.

This is intended to add support for the aws-s3 input function to work in SC2S AWS.

Why is it important?

This is important because the AWS Cloudtrail from S3 integration doesn't currently work in SC2S. It is unable to recognize the region. Not only that, but it has already been added to MetricBeats. Please see /metricbeats/docs/modules/aws.asciidoc line 70.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [~~ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[cla-checker-service]

❌ Author of the following commits did not sign a Contributor Agreement:
e545517

Please, read and sign the above mentioned agreement if you want to contribute to this project

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mgerba? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-04-14T16:23:14.229+0000

• Duration: 5 min 39 sec

Steps errors

Expand to view the steps failures

Load a resource file from a library

• Took 0 min 0 sec . View more details here
• Description: approval-list/elastic/beats.yml

Error signal

• Took 0 min 0 sec . View more details here
• Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/mgerba return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/mgerba : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/mgerba : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[legoguy1000]

Are u saying it currently doesn't work at all? What method are u using, polling or sqs? The function shouldn't prevent functionality, it just sets the provider info, if it isn't in that function it defaults to AWS. Based on the AWS SDK https://github.com/aws/aws-sdk-go/blob/main/aws/endpoints/defaults.go, the sc2s endpoints should be auto discovered and work without issue. What version of filebeat are you using?

[mgerba]

Are u saying it currently doesn't work at all? What method are u using, polling or sqs? The function shouldn't prevent functionality, it just sets the provider info, if it isn't in that function it defaults to AWS. Based on the AWS SDK https://github.com/aws/aws-sdk-go/blob/main/aws/endpoints/defaults.go, the sc2s endpoints should be auto discovered and work without issue. What version of filebeat are you using?

In this case is is polling SQS for s3 file create objects. When the beats input aws-s3 goes to query the SQS queue it throughs a region not found error: "failed to get AWS region from queue_url". The issue is the domain in input.go func getProviderFromDomain.

x-pack/filebeat/input/awss3/input.go

"c2s.sgov.gov":           "aws",
"c2s.ic.gov":             "aws",
"amazonaws.com.cn":       "aws",

The SC2S region is even included in the metric beats docs, but not filbeats.
metricbeat/docs/modules/aws.asciidoc(line 70):
sc2s.sgov.gov
x-pack/metricbeat/module/aws/_meta/docs.asciidoc(line 58):
sc2s.sgov.gov

[legoguy1000]

Are u saying it currently doesn't work at all? What method are u using, polling or sqs? The function shouldn't prevent functionality, it just sets the provider info, if it isn't in that function it defaults to AWS. Based on the AWS SDK https://github.com/aws/aws-sdk-go/blob/main/aws/endpoints/defaults.go, the sc2s endpoints should be auto discovered and work without issue. What version of filebeat are you using?

In this case is is polling SQS for s3 file create objects. When the beats input aws-s3 goes to query the SQS queue it throughs a region not found error: "failed to get AWS region from queue_url". The issue is the domain in input.go func getProviderFromDomain.

x-pack/filebeat/input/awss3/input.go

"c2s.sgov.gov":           "aws",
"c2s.ic.gov":             "aws",
"amazonaws.com.cn":       "aws",

The SC2S region is even included in the metric beats docs, but not filbeats.
metricbeat/docs/modules/aws.asciidoc(line 70):
sc2s.sgov.gov
x-pack/metricbeat/module/aws/_meta/docs.asciidoc(line 58):
sc2s.sgov.gov

That error doesn't relate to the function your referencing. You need to set the endpoint: sc2s.sgov.gov config option.

[legoguy1000]

Did the above solve ur problem??

[mgerba]

Did the above solve ur problem??

The issue with that approach is that is not a configurable option in the AWS CloudTrail from S3 integration. It is possible that it is out of scope for the beats project, but the issue is caused because the input program cannot properly parse the SQS Url.

[legoguy1000]

Did the above solve ur problem??

The issue with that approach is that is not a configurable option in the AWS CloudTrail from S3 integration. It is possible that it is out of scope for the beats project, but the issue is caused because the input program cannot properly parse the SQS Url.

What do you mean? That option is absolutely part of the AWS S3 input. It can't parse the URL properly because by default the FQDN is ..amazonaws.com but for the C2S domains, it is ..sc2s.sgov.gov so it need the endpoint option to know how to parse it.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b main upstream/main
git merge upstream/main
git push upstream main

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b main upstream/main
git merge upstream/main
git push upstream main

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @mgerba? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b main upstream/main
git merge upstream/main
git push upstream main