Load balancing for core service fails

[olevski]

The gateway internally keeps track of the pods for the core service and load balances between them with the addition of providing cookie-based sticky sessions.

But the internal state of the gateway for the core service pods can become outdated and not get updated even though we listen to k8s events.

We can do a few things here:

1. When we see that the gateway cannot reach the core service try to refresh the core service pods IPs in the gateway and try again
2. Investigate how we got out of sync

[olevski]

Related issues on slack:

• https://swiss-data-science.slack.com/archives/CAJMXA6PQ/p1739201855244039
• https://swiss-data-science.slack.com/archives/G016BR3Q9KJ/p1739283099751079