[BUG] failed to list *v1.Secret: stream error when reading response body,...

[laurentiuspurba]

Describe the bug
W1105 20:57:58.830017 1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Secret: stream error when reading response body, may be caused by closed connection. Please retry. Original error: stream error: stream ID 1981; INTERNAL_ERROR; received from peer

To Reproduce
I don't know how to reproduce this. I see it as warning and error in our logs.

Expected behavior
I don't know if this caused the certificate was not renewed because reloader was not reloading

Screenshots
N/A

Environment

• Operator Version: reloader-1.0.121
• Kubernetes Version: 1.30.5-gke.1014001

Additional context
Add any other context about the problem here.

[MuneebAijaz]

@laurentiuspurba how big is the cluster, and its estimated secrets? Also do you face any slowness in listing the resources on the cluster?

[laurentiuspurba]

@laurentiuspurba how big is the cluster, and its estimated secrets? Also do you face any slowness in listing the resources on the cluster?

I have few clusters. In one of the clusters, there were 7000 plus Orders and 7000 CertificateRequests that I had to delete them and then renew the certs; There were around 350 certs in one namespace.

Regarding the slowness, I did experience it when I did a quick kubectl -n THE_NAMESPCE get orders or kubectl -n THE_NAMESPACE get cr. On the other cluster, I need to look into, since my co-worker worked on this, so I did not have the information.

Btw, what are you referring to how big is the cluster? Is it the number of nodes, pods, certs, orders, CR or secerts?

Appreciate your response on this @MuneebAijaz

[MuneebAijaz]

Btw, what are you referring to how big is the cluster? Is it the number of nodes, pods, certs, orders, CR or secerts?

I was refering to the secrets/configmaps/deployments etc, basically any resource that reloader is watching. Since in my experience ive seen this error when there are too many resources which cant be easily cached inside controller cache. I will try to find a solution for it, cant promise anytime soon tho.