Extra Elastic Cloud agent policy agents after upgrading deployment in the cloud

[rdner]

Version: 7.17.19->8.13.0, 8.12.2->8.13.0

Steps to Reproduce:

1. Create a deployment in the cloud
2. Make sure you see an agent running Elastic Cloud agent policy for the current version
3. Upgrade the whole deployment to a newer version (I tried 7.17.19->8.13.0, 8.12.2->8.13.0)
4. Go back to Management->Fleet and you'll see now 2 agents from the Elastic Cloud agent policy: one is the new version, another is from the old version. The old agent gets offline in a few minutes.

Expected behavior

Elastic Cloud agent policy agents are managed, so users cannot manually delete them. Seems like we would have an extra agent on the list per each deployment upgrade and users would not be able to clean them up.

I'd expect that these agents are cleaned up automatically during the upgrade process.

[jlind23]

@rdner I might be wrong but managed agents are docker container hence upgrading the cluster means deploying a new docker image instead of upgrading the current one that's most probably the reason why you see two agents. After some time this offline agent will become inactive anyway and won't show up in the UI.
cc @kpollich

[rdner]

After some time this offline agent will become inactive anyway and won't show up in the UI.

@jlind23 even it's true, I don't think it's good UX. There is no way for the user to clean this up and it will keep piling up with each deployment upgrade.

[kpollich]

Agreed that is is a bad UX. For ephemeral agents specifically (e.g. Docker containers) where the ID changes the old "unenrollment timeout" model worked better than the new "inactivity timeout" model.

@nimarezainia created elastic/kibana#179399 recently that captures the need to support forced unenrollment of inactive agents for specific policies. We could make this unerollment behavior very aggressive for the managed cloud policy if we pursued this. I'm going to close this in favor of the other issue.