Unsigned Requests

[alyssaBiasi]

Currently the following error is thrown when trying to read objects from an S3 bucket with public ACLs:

SdkClientException: Unable to load AWS credentials from any provider in the chain

Would it be possible to add the ability for the SDK to work without credentials?

The CLI does this through the --no-sign-request flag and it looks like something similar will be added to the Ruby SDK.

It would be awesome if the SDK could just work ™️ when there are no credentials available without having to pass it some sort of configuration.

[varunnvs92]

S3 default credential chain has anonymous access which doesn't require any credentials. So you shouldn't have got that exception.
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L412-L425

What SDK version are you using? Can you try with the latest?
If issue still exists, give me sample code to reproduce the issue and wire log.

[alyssaBiasi]

We were on 1.11.106. Just tried 1.11.121 and had the same problem.

Code snippet:

val s3Client = AmazonS3ClientBuilder.standard().withRegion(Regions.AP_SOUTHEAST_2).build()
s3Client.getObject(bucketName, filename).getObjectContent

Error:

SdkClientException: Unable to load AWS credentials from any provider in the chain
	at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:131)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1119)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:759)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:723)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4187)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4134)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1385)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1263)

Edited to remove stray information

[alyssaBiasi]

Just found a second stray set of credentials on my laptop 🤦‍♀️ (sorry this thing hasn't been wiped in 3 years 😬). Edited my last comment to remove the misleading information about access keys.

[alyssaBiasi]

So we did some detective work! 🔍

Because we're creating the S3 client using the AmazonS3ClientBuilder our client is never constructed with the internal S3CredentialsProviderChain, just a DefaultAWSCredentialsProviderChain.

We found that we can get around this by doing the following (excuse the hacky scala 🙈):

val s3Client = AmazonS3ClientBuilder.standard()
                                    .withRegion(Regions.AP_SOUTHEAST_2)
                                    .withCredentials(new MyCreds)
                                    .build()

class MyCreds extends DefaultAWSCredentialsProviderChain {
    override def getCredentials = {
      try
        super.getCredentials
      catch {
        case ace: AmazonClientException => {
        }
      }

      null
    }
  }

Should I open a new issue for this or re-purpose the current one?

(Sorry for not including the wire log. We haven't figure out how to get it working with our scala app)

[varunnvs92]

The SDK should be using S3CredentialsProviderChain instead of DefaultAWSCredentialsProviderChain for S3. This looks like a bug in the SDK. Thank you for the research. I will look into it and update you soon.

[varunnvs92]

The fix will be included in our next release

[kiiadi]

@alyssaBiasi looking at this a bit more when null is returned by the AWSCredentialProviderChain rather than an exception - this has the effect of making S3 requests 'annoymous'. Rather than silently falling back to this behaviour it probably makes sense for customers to 'opt-in' to it by providing their own credential chain (as you've done). We could even add this class as part of the SDK as a helper something like AWSAllowAnnoymousCredentialProviderChain.

I realize that the S3 builder is now inconsistent with the client constructor however, the builders represent how we intend the code to behave in the future. It doesn't really make sense for S3 to have it's own credential provider chain just to support this case that is easily configurable. The S3 client constructor method also behaves inconsistently with other clients making debugging more complicated - compounding the desire to move away from the custom chain.

Thoughts about adding an AWSAllowAnnoymousCredentialProviderChain to the SDK and having this be configurable rather than the default? Would this satisfy your use-case?

[alyssaBiasi]

I think calling out the fact that the request falls back to anonymous makes sense to me @kiiadi. We ended up calling our one AnonymousFallbackCredentialProviderChain (names are hard) so what you're suggesting would work for us 🙂

[kiiadi]

@alyssaBiasi since we've already corrected the behaviour we won't change this for the current version of the SDK - however in the next major version we'll likely move to the approach I described above.

Closing this for now since the change to make it the builder consistent with the constructor has already gone out in 1.11.123 of the SDK last wee.

[cipri-tom]

Hello,
I have stumbled upon this issue in March and couldn't debug it myself :(. I was happy to see new version coming out, but it seems there is still no fix ?

I want to access public datasets from Spark which relies on aws-java-sdk. I tried setting the fs.s3a.aws.credentials.provider in Hadoop to AnonymousAWSCredentialsProvider but it obviously fails. The cause (same as here) was pointed out in this issue on StackOverflow : it is not allowed (by design?).

Sorry for being a noob, but could you please point out how I can have anonymous access ? Or it's simply not possible with current version ?

[shorea]

From the issue it appears to be a bug in Spark, the SDK definitely allows anonymous access for S3 (default fallback behavior in both client constructors and per this issue the client builders). Can you move this issue to Apache Sparks repo?

[cipri-tom]

@shorea thank you for the timely reply !

I submitted an issue to Apache Spark too.

But to me the issue seems to come from these lines in AWSCredentialsProviderChain along with the return null for AnonymousAWSCredentials

I am using aws-java-sdk-1.7.4.jar and I discovered the lines because I checked if they changed in v1.11. They didn't.

P.S. I am commenting here rather than new issue because it fails with the same message

[itudoben]

Hi,
a year later I'm facing the same issue with public S3 access, and from the code links from @cipri-tom and I double checked it, this seems that by design anonymous - public access ? - won't ever work.

Is there any AWS person to confirm or refute this is the case?

I'm stuck so I'll figure out something else for now but it would be great to get to the bottom of this issue.

Thx all.