Workflow Pods intermittently fails with 'Error (exit code 1): Unable to connect to the server: dial tcp 10.100.0.1:443: i/o timeout' message

[Sushant20]

AWS EKS Kubernetes version : 1.19
Argo workflows version: 3.1
Errors are visible in 'main' container, it seems related to #4993

Will configuring executor retries help?
https://github.com/argoproj/argo-workflows/blob/master/docs/environment-variables.md

Would also like to understand why are such intermittent errors happening?

[sarabala1979]

The timeout is coming from k8s API server. it could be many reason. 1. API limit. 2. load on your cluster