[Kubernetes Integration] Implement Rate Limit mechanism for Kubernetes API calls

[gizas]

Describe the enhancement:

Kubernetes Integration performs API calls to the Kubernetes API in order to retrieve information for the resources it monitors .It uses the provided information to enrich the metadata of the documents that ingests. The implementation is based on client-go library and this is an example of a watcher that will start monitor namespace events and performs API calls in the background. The mechanism of API calls should have a way to limit the number of calls towards KubeAPi to prevent the exhaustion of the server.

Cases of 429 Too many requets from APi server, should be handled gardually from the code, intially to limit/stop the requests when error occurs (or if possible if in big delays of responses) and perform a retry on a given time period.

Describe a specific use case for the enhancement or feature:

We have seen issues (3991, 4325) that customers complain for the large number of API calls beats or elastic agent performs to the KubeAPI, especially in big scale clusters.

Related Links

• https://kubernetes.io/docs/concepts/cluster-administration/flow-control/
• https://github.com/kubernetes/client-go/blob/master/util/flowcontrol/backoff_test.go
• https://github.com/cenkalti/backoff
• https://pkg.go.dev/golang.org/x/time/rate#pkg-index

[anson627]

we found API server OOM killed where beats list the resources nodes/namespaces/replicasets from the daemonset pods from all nodes, is there any plan to move forward with this issue?