Fleet Server returns 500 on serverless causing elastic-agent to go offline

[olegsu]

Steps to reproduce

• Deploy serverless project
• Enroll elastic agent
• Follow the fleet server logs

I noticed that fleet server returns 500 sporadically, causing the elastic agent to go offline.
More logs we can find the monitoring cluster

Related

• https://github.com/elastic/security-team/issues/6701

[michel-laterman]

I'm not sure what we can do about this.

It looks like fleet-server had an issue querying ES and got a 500, fleet-server returned a 500 as a result, we can try to return a 503 to indicate it's a temporary failure.
If ES is failing there is not a lot we can do, if ES rejects queries fleet-server will not be able to process checkins so after a little bit the UI will show agents as offline.

@kpollich what do you think?

[kpollich]

Yeah there's not really anything we can do when Elasticsearch returns a 500. Responding a 503 in this instance is probably the best we can do, but from Fleet Server's perspective a core dependent service (Elasticsearch) is unreachable at the time of this request. A better path forward might be determining why Elasticsearch is throwing a shard error and a 500 in this particular instance.

[olegsu]

Thank you for the updates @michel-laterman, @kpollich
We workaround this by adding init-container to wait for fleet ready