Keep the AWS/Azure/GCP SDK version up to date

[zmoog]

Situation

All major CSPs provide SDKs for accessing their API. Beats use the cloud provider API to collect logs, metrics, and metadata.

Over time, CSPs release new versions of SDK. Most of the time, new versions are minor or patch releases. Major releases with breaking changes usually happen every few years.

Usually, we upgrade the cloud provider SDK in two circumstances:

• we need a new feature
• we need a bugfix

That's only available in a new SDK version.

Our attitude is primarily reactive.

Problem statement

The current reactive posture has a few downsides:

• On average, our SDK modules are outdated to various degrees (missing fixes)
• Upgrades happen not so often, sometimes taking big jumps in versions (increasing risks)
• On average, if we need a bugfix in one of our dependencies, we can add it now, but the subsequent stack releases may be weeks away; we only backport to the previous release.

Solutions

Manage AWS/Azure/GCP SDK version incrementally using Dependabot.

Pros:

• It is more manageable to upgrade 1-2 dependencies at a time instead of doing a big batch occasionally.
• SDKs are up to date with fixes and improvements
• We integrate updates in the next release to improve the change or avoid bug reports and support requests.

Cons:

• We are making more changes; we need to mitigate this risk by improving our test suite (we'll address this in a dedicated issue).

Tasks

Beta Give feedback

1. Manage Azure SDKs version with Dependabot #39495
   2 of 2
   Team:obs-ds-hosted-services +1
   
2. Manage GCP SDKs version with Dependabot #39739
   Team:obs-ds-hosted-services +1
   
3. Dependabot: add CSP SDKs to the allow list #40150
   Team:obs-ds-hosted-services bug +2
   
4. Manage AWS SDK(s) version with Dependabot #39738
   Team:obs-ds-hosted-services +1
   

Related

Here are a few related issues and PRs:

• Fix the AWS SDK dependencies issue causing the "not found, ResolveEndpointV2" error #39454

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[zmoog]

Here is how APM Server manages OTel dependencies:
https://github.com/elastic/apm-server/blob/main/.github/dependabot.yml

[zmoog]

I'm creating the first PR to manage the Azure SDK to get feedback from the team responsible for Dependabot.

[agithomas]

If we notice the issue mentioned here, a sequential upgrade from working -> broken SDK (Nov '23 -> Feb'24) , it would have taken the application into a less desirable state.

So, the scope may be extended or changed to qualifying the SDK version. Addressing dependencies, Testing & verification, addressing gaps in testing (if any) may be part of the qualification process. Once qualified, how soon should we consume the upgrade?

[zmoog]

If we notice the issue mentioned here, a sequential upgrade from working -> broken SDK (Nov '23 -> Feb'24) , it would have taken the application into a less desirable state.

So, the scope may be extended or changed to qualifying the SDK version. Addressing dependencies, Testing & verification, addressing gaps in testing (if any) may be part of the qualification process. Once qualified, how soon should we consume the upgrade?

@agithomas, what do you mean by "qualifying the SDK version"? Can you elaborate a little on this?

[zmoog]

To be clear, the goal of PRs like #39495 is not to avoid problems like "not found, ResolveEndpointV2" but to ensure Beats remains up to date with new features, possibly avoid support requests, or at least ship the fixes sooner.

I agree that to avoid problems like "not found, ResolveEndpointV2" from happening we need a holistic strategy that includes:

• improved dependency management
• improved testing

[agithomas]

@agithomas, what do you mean by "qualifying the SDK version"? Can you elaborate a little on this?

My comment was more towards the holistic strategy.

Thanks for adding the scope here.

[zmoog]

After the last fix #40125, the config is valid, but the current Dependabot config is still not okay.

From https://github.com/elastic/beats/network/updates/852878485

updater | 2024/07/08 15:25:07 WARN <job_852878485> Please check your configuration as there are groups where no dependencies match:
updater | - azure-sdks
updater | - gcp-sdks
updater | 
updater | This can happen if:
updater | - the group's 'pattern' rules are misspelled
updater | - your configuration's 'allow' rules do not permit any of the dependencies that match the group
updater | - the dependencies that match the group rules have been removed from your project
updater | 
updater | 2024/07/08 15:25:08 INFO <job_852878485> Starting grouped update job for elastic/beats
updater | 2024/07/08 15:25:08 INFO <job_852878485> Found 2 group(s).
updater | 2024/07/08 15:25:08 WARN <job_852878485> Skipping update group for 'azure-sdks' as it does not match any allowed dependencies.
updater | 2024/07/08 15:25:08 WARN <job_852878485> Skipping update group for 'gcp-sdks' as it does not match any allowed dependencies.

[zmoog]

The scope of this issue was adopting Dependabot to manage the CSP SDKs dependencies.

I am taking #39505 out for next iteration.