Memoizing and caching CredentialsProviders return credentials that are just about to expire

[ho-obviux]

My PHP application uses aws-sdk-php v. 3.61.5 to publish messages to an sns topic. The application is running on an ec2 instance with an instance profile that allows it to publish messages to the sns topic.

The application is supposed to publish around 600,000 messages to the SNS topic as fast as it can. Everything works well for the first several hundreds of thousands messages. After several hours, however, it fails with this message:

Aws\Sns\Exception\SnsException

Error executing "Publish" on "https://sns.eu-west-1.amazonaws.com"; AWS HTTP error: Client error: `POST https://sns.eu-west-1.amazonaws.com` resulted in a `403 Forbidden` response:
<ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
  <Error>
    <Type>Sender</Type>
    <Code>ExpiredToke (truncated...)
 ExpiredToken (client): The security token included in the request is expired - <ErrorResponse xmlns="http://sns.amazonaws.com/doc/2010-03-31/">
  <Error>
    <Type>Sender</Type>
    <Code>ExpiredToken</Code>
    <Message>The security token included in the request is expired</Message>
  </Error>
  <RequestId>e6c59e4f-b998-5d99-b926-cafe6ac2711a</RequestId>
</ErrorResponse>

So clearly the security token expired.

I am using the following code to create the sns client:

        $provider = CredentialProvider::defaultProvider();
        $provider = CredentialProvider::cache(
            $provider,
            new DoctrineCacheAdapter(new FilesystemCache('/tmp/cache'))
        );

        $this->client = new SnsClient([
            'region'      => 'eu-west-1',
            'version'     => 'latest',
            'credentials' => $provider
        ]);

The caching CredentialsProvider will ask the underlying (default) CredentialsProvider for credentials if the currently cached credentials are expired. And the default CredentialsProvider will refresh the credentials if they are expired.

The problem, as I see it, is that both the caching CredentialsProvider and the default CredentialsProvider continue to return the existing credentials up to the very last millisecond where they are not expired. This means that just before they expire, the sns client will fetch credentials and use them to call the sns service. By the time the request reaches the sns service, the credentials have expired, and the service consequently rejects the request.

According to AWS docs (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials) refreshed credentials are available from the instance metadata at least five minutes before they expire, so perhaps it would be better for the CredentialsProvider implementations to refresh the token when it expires in less than five minutes?

If I am not using the sdk correctly, some guidance would be highly appreciated :-)

[veihmeyer]

nice

[kaktusas2598]

We are having same issue, except that in our case it is mainly s3:GetObject operation. We use memoize and chain method of CredentialProvider class. I don't why this is marked as feature request, it looks like a bug to me, because we took this snippet from official AWS SDK Documentation: https://docs.aws.amazon.com/sdk-for-php/v3/developer-guide/guide_credentials_provider.html#chaining-providers