S3 costs due to authentication error

[creinheimer]

Hi there,

Over the last few days, we experienced an issue with our AWS IAM role configuration for service accounts on Kubernetes. This caused the Loki pods to fail authentication with S3, resulting in increased S3 costs due to many AWS PutObject calls.

I will try to update our Loki clusters later to a recent version, as this #14174 was introduced in Loki 3.1.x.

Anyhow, I thought it was relevant to open an issue in case someone comes across the same situation.

Expected behavior
Retry exhaust or any solution that doesn't lead to high costs.

Environment:

• Infrastructure: Kubernetes 1.31, EKS, AWS, S3
• Deployment tool: Helm chart 5.37.0
• Loki version: 2.92

Screenshots, Promtail config, or terminal output

S3 Errors 500 PutObject