Monokle policies as CRDs

[f1ames]

This PR fixes https://github.com/kubeshop/monokle-saas/issues/1839, fixes https://github.com/kubeshop/monokle-saas/issues/1845.

Implemented MonoklePolicy and MonoklePolicyBinding according to https://github.com/kubeshop/monokle-saas/issues/1839#issuecomment-1738937312 and consistent (to some extend) with how VAP works.

Changes

• Introduced CRDs for MonoklePolicy and MonoklePolicyBinding.
  • VAP supports 3 action types. For the first version we decided to go with Warn only (see screenshot below). I already put it in the CRDs definition (as enum array so it can be extended and is the same format VAP has).
• Introduced PolicyManager class which watches instances of MonoklePolicy and MonoklePolicyBinding and is responsible for storing, updating and matching (policy-binding-resource) them (I think *Manager class name is a bit of naming-antipattern so I'm open for suggestions).
  • This is based on K8s Informer (high level watcher) mechanism.
• Introduced ValidatorManager class which reacts to policy changes and creates/updates/deletes validator instances. It also provides matching validators for given resources when they need to be validated.
• Reworked the structure of the project. Also, by default all namespaced resources related to admission controller deployment will be deployed to monokle-admission-controller namespace.
• Added mechanism blocking plugins which will not work at the current state (resource-links). See https://github.com/kubeshop/monokle-saas/issues/1845#issuecomment-1744412694 and 586aeed.

Sample output (we can adjust wording there):

Fixes

• Mostly major changes as described above.

To consider

• Namespace matching - see this comment.
• Revisit logs - I used pino for logging and try to make it reasonable with different levels (so it's not to much with info and higher level while debug/trace will log a lot of data on each update, like resources content, polices content, etc). Still it might require some improvements to better help with debugging.
• Deployment part (via Helm) is not there yet since we have separate task for it - https://github.com/kubeshop/monokle-saas/issues/1842.

I'm open for discussion/suggestions for any of the above.

Testing

See README.md with step-by-step instructions.

Checklist

• tested locally
• added new dependencies
• updated the docs
• added a test

[murarustefaan]

I really think we should reconsider our approach on namespaced admission controllers. Or maybe i did not understand the code and I might be wrong, but the approach where the controller can have an one "active policy" at a time is not looking reliable to me.

A better aproach, on a quick thought, would be creating a Map, 1-1 between a namespace and a policy (at first, later we can map resources to policies better should we want that and if it is possible). Now the Controller would have knowledge of all the policies and would be able to validate namespaced manifests.

[olensmar]

I really think we should reconsider our approach on namespaced admission controllers. Or maybe i did not understand the code and I might be wrong, but the approach where the controller can have an one "active policy" at a time is not looking reliable to me.

A better aproach, on a quick thought, would be creating a Map, 1-1 between a namespace and a policy (at first, later we can map resources to policies better should we want that and if it is possible). Now the Controller would have knowledge of all the policies and would be able to validate namespaced manifests.

totally agree - this would be configured in cloud, i.e. which policies apply to which resources - the selection could be based on

• namespace
• labels/annotations
• resource kinds
• ?

[f1ames]

Ok, discussed some technicalities with @murarustefaan to make sure I understand how admission controller should behave exactly, to summarize:

• Admission controller should be global (not namespaced) and should react on resources creation/updates across namespaces (this will be configurable later, which namespaces, etc).
  • Still, webhook deployment/pod will belong to a namespace (should be configurable during deployment).
• Based on resource data, it will use policy from the same namespace to validate it.
  • Should we fallback to any default if there is no policy in a given namespace?
• When integrating with Monokle Cloud, it should allow configurabilty, to define which policies should be applied to which resources (see Monokle policies as CRDs #1 (comment)).
  • I already added some initial thoughts on configurability here - https://github.com/kubeshop/monokle-saas/issues/1839#issuecomment-1735503488. Suggestions welcome 👍

[f1ames]

I'm not entirely sure about this part. It follow what VAP has but it's labels matcher. Initially I thought we would have a property to define namespace name for which policy is applied (through this binding). This one matches namespace label in resource metadata which is not required and can have any value AFAIU.

I went through the specs but haven't found something which would just match namespace for the resource 🤔 Not sure if there is something from the specs we can easily use or should have something custom 🤔

[olensmar]

also need to be mindful that no all resources are namespaced - is it possible to match on resource kind/version?

[f1ames]

This is optional part of MonoklePolicyBinding so skipping it will not restrict policy to be applied to single namespace (so will be applied cluster-wide).

As for more precise matching, we can follow matchResources spec which allows filtering by resource, apiVersion, apiGroups, etc, with resourceRules. By "following" I mean we need to implement matching mechanism according to specs (or part of it) in admission controller.

[WitoDelnat]

This is such a common use case that Kubernetes automatically adds a label to all namespaces called kubernetes.io/metadata.name=$namespaceName.

[f1ames]

See kubeshop/monokle-core#538.

[f1ames]

See this comment. This is a bit "stretched" interpretation of namespace matching (when resource doesn't have metadata.label.namespace). But I have some doubts about it.

[WitoDelnat]

The link is broken. Not sure I understand the problem correctly. For normal Kubernetes Objects you would look to metadata.namespace because metadata.label.namespace will rarely exist.

[f1ames]

The link lead to my previous comment which you already answered with:

This is such a common use case that Kubernetes automatically adds a label to all namespaces called kubernetes.io/metadata.name=$namespaceName.

So my understanding is that below matching which we want to support:

  matchResources:
    namespaceSelector:
      matchLabels:
        namespace: default

is metadata.label matcher according to specs. Meaning being compliant with specs, it should only try to match resource.metadata.label.namespace?

Or maybe not 🤔 It doesn't really say how matching is done for Namespaced resources:

NamespaceSelector decides whether to run the admission control policy on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the policy.

Not sure now TBH 🤔

[WitoDelnat]

It's a 2-way lookup:

• Give me the namespace that match these labels.
• Match me all the resources that match this namespace selector.

You can see it in action in the excerpt below from this blogpost.

const namespaces = matchExpressions(allnamespaces, predicates); // Returns many namespaces which are not these system namespaces
const objects = matchResources(namespaces); // Returns all namespaces looking at `meta.namespace`

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-deployments
spec:
  paramRef:
    name: max-replicas-deployments
    namespace: policies-configs
    parameterNotFoundAction: Deny
  policyName: max-replicas-deployments
  validationActions:
    - Deny
  matchResources:
    namespaceSelector:
      matchExpressions:
      - key: kubernetes.io/metadata.name
        operator: NotIn
        values:
        - kube-node-lease
        - kube-public
        - kube-system
        - allow-listed

I'd say let's not dive too deep into this yet. It will likely need to be a library which both the ValidatingAdmissionPolicy validator and the Admission Controller use. In this library, we can cover everything thoroughly with unit tests as the expressions can get quite elaborate. If we simply cover this exact example and maybe one with operator In so you can specify in namespace demo we are good to go. Can even be in a follow-up or bootstrap that library as now we can already apply the policy globally and these system namespaces can be ignored with config iirc.

[f1ames]

@WitoDelnat thanks for the explanation, now I understand👍 Well, not it works fine with matching resources in given namespace (thanks to checking metadata.namespace as you described) and selector matching. So it seems to be consistent with what you described.

For more general handling of matchResources with matchExpressions, resourceRules and others I will add an issue in monokle-core as you suggested 👍

[f1ames]

Reported as kubeshop/monokle-core#538.

[WitoDelnat]

Splendid work 💪