AWS authentication conventions not fully implemented

[cpinflux]

Steps to reproduce:
List the minimal actions needed to reproduce the behaviour.

1. Log in to AWS by cli: aws sso login
2. Run influxdb3 with an s3 bucket

Expected behaviour:
Describe what you expected to happen.

influxdb3 should integrate with the AWS SDK for authentication and authenticate based on the previous AWS login, without needing to export the standard AWS environmental variables of AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Actual behaviour:
Describe What actually happened.

Service fails with authentication issues

Additional Information:
The desired behaviour is seen in the aws cli client itself and also, for example, terraform. Neither require the environment variables to be exported in order to utilise the existing AWS session. This also means that authentication is not limited to the scope of a single bash session, an aws login in one terminal results in (for example) terraform being able to access AWS from other terminals too.

This is the same behaviour I observed with datafusion-cli, as suggested by @alamb, suggesting that there is a common root cause here, presumably in object_store, as influxdb3 uses object_store's AmazonS3Builder interface to establish connectivity S3.

[alamb]

FWIW the code that uses the native ASK SDK to get authentication information in datafusio-cli is here:

https://github.com/apache/datafusion/blob/2aff98e002ce6d48008b8bbe2b38ee644a6d5c20/datafusion-cli/src/object_storage.rs#L112-L133

(that isn't included directly in object_store as the official AWS SDK is huge and has many dependencies)

[cpinflux]

I believe I have established that the authentication process that is working in this scenario is either "Assume role" or "Assume role with web identity" here https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-authentication.html and under the hood ~/.aws/cli/cache/xxxxx.json is being used. This is a newer file than the previously used ~/.aws/credentials and there are a number of issues I see in the past relating to this on Terraform where support was officially added in the go api. aws/aws-sdk-go#3755 Naturally this is go, not rust, but potentially points towards equivalent functionality hopefully already present in the rust SDK.