Inconsistent health.status in Kargo from what is shown in Argo

[NathanHRich]

Checklist

• I've searched the issue queue to verify this is not a duplicate bug report.
• I've included steps to reproduce the bug.
• I've pasted the output of kargo version.
• I've pasted logs, if applicable.

Description

Kargo stages seem to be inconsistent with what constitutes "Healthy" and "Unhealthy" health.status
In the attached screenshots, the first is reporting "Unhealthy" because the Argo CD Application is out of sync. Then Healthy even when the syncStatus.status is still OutOfSync.
In both of these cases Argo CD shows the Application as "Healthy", which is what I expect to be reflected in Kargo.

Screenshots

Steps to Reproduce

Unfortunately this just pops up sometimes... I'm not sure how to reproduce it.

Version

Running in a kubernetes cluster along side argocd
The screenshots were taken in v0.8.5
We just upgraded to v0.8.7, I'll confirm here if I see this pop up in v0.8.7

[krancour]

This isn't a bug. This is because Stage health and App health are not the same thing.

App health factors into Stage health, but they are not the same.

Consider the following:

• Some Stages may not have any Apps at all.

• Some Stages may have multiple Apps.

• An App that is out sync in Argo CD can be "normal," as believe it or not, many Apps exist in a perpetual state of drift.

• An App that is in sync in Argo CD may still be problematic from Kargo's perspective if it is synced to the wrong revision.

• Kargo may integrate well with Argo CD, but it would be short sighted of us if it were going to work only with Argo CD forever. Our plans are bigger than that. Consider especially that we are working hard to make Kargo extensible, which means any number of third-party integrations are on the horizon and they may all contribute to the health equation in different ways.

Taken all together, Kargo Stage health is not the same as App health. What it is is as comprehensive a snapshot of how the Stage is doing at the moment as we are able to build. We are always trying to improve it, but at the moment, if you are using it with Argo CD, a Stage is going to appear healthy if all related Apps are Healthy, synced to the correct revision(s), and done syncing, even if they report being out of sync. i.e. Apps being done syncing to the right things + being healthy makes a Stage healthy.

Now, with all of that being said...

It's sometimes a bit complicated to make this assessment and there have been plenty of bugs related to it that we have had to fix and it wouldn't shock me if there are more. So if, with the above explanation in mind, something still doesn't seem right to you, more detail might lead us to something actionable, however, Stage and App health merely not matching isn't necessarily a sign of something amiss.