How to "Scope" cluster singleton operators to specific namespaces

[awgreene]

An inherent goal of RukPak is to treat operators as cluster scoped singletons, implying that there will not be multiple versions or installations of an operator on a cluster. There have been requests to support "scoping" theses cluster singleton operators to events in specific namespaces through the use of RBAC. If "scoping" is supported through the use of RBAC, we must consider how operators will configure their informers to watch for events in namespaces where they have appropriate RBAC permissions. Failure to do so could cause requests against objects found in the informer to fail due to missing permissions.

The goal of this discussion is to track this requirement and foster conversation.

[joelanford]

There are many schools of thought on how much access operators should have throughout the cluster. I think I've landed at:

1. Operators should always have a cluster role and cluster role binding for get/list/watch on their primary resource and update on their primary resource's status subresource throughout the cluster. That way, regardless of the operator's permission to manage operands, they can at least update status to say "sorry, i don't have permission to manage this resource in this namespace"
2. Ideally, operators should attempt to use cluster-scoped informers for their watches because they are less resource-intensive that many individual namespace-scoped caches. In the event the operator does not have permission to watch at the cluster scope, it should fallback to dynamic multi-namespace caches, which would basically spin up and down as primary objects came and went.
3. If the dynamic multi-namespace cache fails to spin up a namespace-scoped cache in a namespace in which a primary object exists or if the operator doesn't have write permission on secondary operand resources, then the operator should (via 1) update the status of that object to reflect the lack of permission.

The dynamic multi-namespace cache (DMNC) was suggested in kubernetes-sigs/controller-runtime#1590 and should be revisited if we think this idea is worth pursuing. It would also be interesting to investigate a cache implementation that attempts cluster-scoped watches and falls back to the DMNC.