Endless loop of heartbeat stopped-heartbeat-restored events if an endpoint is re-deployed with same HostId but different Host

[SzymonPobiega]

Describe the bug

Description

If an endpoint is stopped and re-deployed with a changed Host but the same HostId settings, ServiceControl heartbeat state machine creates duplicated state for that instance. One copy shows the endpoint instanced as dead and another as alive.

Both copies of state are fed into the state machine every time ServiceControl attempts to detect dead endpoints leading to flipping between alive and dead on each attempt.

Expected behavior

The old instance shows as dead/inactive and can be removed from the ServiceControl settings. The new instance shows as active.

Actual behavior

ServiceControl shows a single instance that constantly flips from active to inactive

Versions

The bug has been detected in the latest version of ServiceControl but likely affects all versions.

Steps to reproduce

Manually override the host name and host id.

Relevant log output

No response

Additional Information

Workarounds

Not possible

Possible solutions

Use full EndpointInstanceId hash code as a key in the endpoints collection of EndpointInstanceMonitoring.

Additional information

Normally it is very rare to change the Host but not HostId because the default strategy for generating HostId takes into account the Host field. This issue has been discovered because of Particular/NServiceBus#7026 -- a bug in NServiceBus host name override API.