Allow opt-out to namespace deletion when Project is deleted

[ddeath]

Checklist

• I've searched the issue queue to verify this is not a duplicate feature request.
• I've pasted the output of kargo version, if applicable.
• I've pasted logs, if applicable.

Proposed Feature

I would like to have an option to say Kargo that I do not want it to delete namespace when kargo Project is deleted.

Motivation

I noticed that there is option to use existing namespaces: #2528 however when I used it and then I deleted Kargo Project half of my cluster went missing :/

Because Kargo is creating namespaces based on Project names there is higher chance there will be some conflicts. Because if Kargo Project is used to represent Bounded Context in organization there is high chance we already have namespace with name of that Bounded Context.

Suggested Implementation

• could be an option in helm chart which would be rendered to some Kargo configmap
• adding label / annotation indicating the namespace should not be deleted

[krancour]

I think there are some difficult security implications to this.

I do think this is a good idea, but we need to be very careful about how it is implemented.

On the security front, what you would not want is for someone to delete a Project, have the namespace stick around, and then have someone add a new Project with the same name, have it adopt that namespace, and this would effectively permit a new Project to hijack and old one, including credentials/Secrets. We would probably have to do this similar to what k8s does with PVs/PVCs and "reclaim policies."

Unrelated to that, we would also have to figure out an efficient way of preventing reconciliation of Kargo resources living in a namespace that no longer has a Project associated with it.

[ddeath]

I dont have experience with this too much, but my expectation is that Kargo will delete only things created by itself? Then it should be kind matter of just labeling / annotating resources? I think ArgoCD does the same.

I have probably very simplified view about this again 😄 and in reality it will not be that simple 🤷

[krancour]

my expectation is that Kargo will delete only things created by itself?

I'm not sure I see how this is relevant to the things I said we'd have to watch out for if we were to implement this.

[ddeath]

I'm not sure I see how this is relevant to the things I said we'd have to watch out for if we were to implement this.

Well if you delete Project then everything created by Kargo would be gone right? That means there is nothing to be hijacked because all connected service accounts, all secrets and other stuff would be deleted with Project 🤷

[ddeath]

I would make it an option then. Option to indicate that NS should be deleted no matter what it contains

[krancour]

Well if you delete Project then everything created by Kargo would be gone right?

Ah. I didn't understand this to be what you were asking for.

I assumed that in asking for (the option for) a namespace to not be deleted, you'd also want to preserve what's in the namespace.

What exactly would be the point of not deleting the namespace if your expectation is for Kargo to manually clean up everything within it?

[ddeath]

What exactly would be the point of not deleting the namespace if your expectation is for Kargo to manually clean up everything within it?

not everything, just things created by Kargo itself.

The point is that you can share namespace with your other applications.

Lets say you have onboarding namespace which contains all onboarding related microservices and you want to have Kargo Project called onboarding. Currently you cant do that because you risk that Kargo will delete all your microservices...

The only option is to move cargo to separate cluster, if we want to do that we need to move argocd as well because that is Kargo requirement....

[krancour]

not everything, just things created by Kargo itself.

Kargo creates Freight. It might create Promotions if you have auto-promotion enabled for some Stages. Everything else in a Project is created by you. That includes Warerhouses, Stages, and most importantly, Secrets. If Kargo cleans up things it created, Secrets remain and my concern about hijacking the namespace remains as well. If you have actual application resources, including Secrets in that namespace, my concern about hijacking actually increases.

The point is that you can share namespace with your other applications.

Lets say you have onboarding namespace which contains all onboarding related microservices and you want to have Kargo Project called onboarding.

We absolutely, positively did not intend for people to have their Kargo resources and actual application resources in the same namespace. Please don't do that. That wouldn't even be an option in a lot of cases, because Kargo isn't always on the same cluster as your application(s). Even in the case where it's possible, doing so is going to massively complicate using RBAC for managing access to Kargo and/or application resources in that namespace.

The only option is to move cargo to separate cluster

This is actually common and it is this way by design. Kargo resources live in a common Kargo control plane that is visible to Kargo controllers running in many different clusters.

if we want to do that we need to move argocd as well because that is Kargo requirement.

This is also not true. A single Kargo controller can live in any cluster and can interface with one Kargo control plane and one Argo CD control plane. Either or both of those things can be remote from the perspective of the controller.

Admittedly, this hasn't been well-documented yet.

I'm going to close this issue because now that I understand more clearly what you are asking for, it's not something that we're going to consider.

If you need more help understanding topological issues, please raise the question in the Discord community or browse other issues to see there has been a lot of prior Q and A about how how the pieces all connect.