[Self-Managed]: Fleet Server doesn't gets back healthy itself when setup VM is turned off then re-run after sometime.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.11.0 SNAPSHOT self-managed
BUILD: 67254
COMMIT: 7ac96504f11ebde1f14d7cd9d45d99ed0d81768f

Host OS: Windows

Preconditions:

1. 8.11.0 SNAPSHOT Kibana self-managed environment should be available.

Steps to reproduce:

1. Setup 8.11 self-managed elasticsearch and kibana.
2. Install Fleet Server.
3. Shut down elasticsearch and kibana services. Now shut down the VM
4. Turn VM on after 10 minutes and also re-run elasticsearch and kibana again.
5. Navigate to Fleet tab and observe Fleet server keeps sending data and logs, however remain offline on UI.
   [We have waited more than an hour to get the fleet server back Healthy.]
6. Agent diagnostics also fails.
7. Now run .\elastic-agent.exe diagnostics collect.
8. Observe agent gets back Healthy.

Note:

• We have observed this issue on earlier kibana versions too.
• Issue is consistent and reproducible everytime.

Agent logs:
elastic-agent-diagnostics-2023-09-25T10-35-16Z-00.zip

Screen Recording:

Amol.Self-Win-ON.1.-.ec2-18-207-154-190.compute-1.amazonaws.com.-.Remote.Desktop.Connection.2023-09-25.16-50-57.mp4

Expected Result:
Fleet Server should get back healthy itself when setup VM is turned off then re-run after sometime.

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Secondary review for this ticket is Done