Intermittent InternalError on PutObject requests

[litsters]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Our electron app ([email protected]) uses the SDK to copy a large number of image files (hundreds to a thousand in a set, 120 sets a day) from a local disk to our S3 bucket. We recently switched from version 2 of the SDK to version 3, and now we are getting a large number of 500 errors from PutObject requests. Prior to the switch we had never had one of these errors surface, and now the majority of sets we try to upload have one interrupt the process.

Errors are hard to reproduce, as when we tell the system to retry these failures it's usually able to do them without a problem. The full error is InternalError: We encountered an internal error. Please try again.

We use anonymous clients for these uploads, so we can't use the version 3 Upload to do this. We stream the files from disk to avoid reading all of them into memory, and do about 40 PutObject requests at a time.

I was able to get a screenshot with details about the request itself. Hoping that will make this easier to look into.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

16.17

Reproduction Steps

These errors aren't deliberately reproducible. Whether or not they happen is random and the error is transient.

Observed Behavior

We see a lot of PutObjectCommands fail with S3 putObject failed with S3ServiceException [InternalError]: We encountered an internal error. Please try again.

Expected Behavior

I expect these errors to not be thrown, or at least be thrown rarely. The message suggests they can be retried, but they're being thrown anyways.

Possible Solution

Maybe the version 3 sdk has changed how it handles stream uploads so that automatic retries don't happen.

Additional Information/Context

No response

[aBurmeseDev]

Hi @litsters - thanks for reaching out.

The error you're seeing is a generic error message from S3 indicating that there was an unexpected issue on the AWS side during the PutObject request. This error is typically transient and retrying the request usually resolves the issue.

Are you retrying these requests? If so, consider increasing timeouts values. Sometimes, transient issues can cause requests to take longer than expected leading to timeouts and errors.

Another recommendation is to consider a retry mechanism with exponential backoff for failed PutObject requests. This will help reduce the load on S3 by spacing out retries and giving AWS time to recover from any temporary issues.

Here's docs on retry strategy and backoff computation: https://github.com/smithy-lang/smithy-typescript/blob/main/packages/util-retry/README.md

If issue persists, please share a minimal reproducible code for further guidance and investigation.

Hope it helps!
Best,
John

[litsters]

Thanks for the response. It does look like timeouts may be a factor here, so I'll look closer there.

[aBurmeseDev]

Any luck or updates on getting it resolved?

[github-actions]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.