Add Cluster API deployment method for TAS

telemetry-aware-scheduling/deploy/cluster-api/README.md

@@ -0,0 +1,17 @@
+# Cluster API deployment
+
+## Introduction
+
+Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. [Learn more](https://cluster-api.sigs.k8s.io/introduction.html).
+
+This folder contains an automated and declarative way of deploying the Telemetry Aware Scheduler using Cluster API. We will make use of the [ClusterResourceSet feature](https://cluster-api.sigs.k8s.io/tasks/experimental-features/cluster-resource-set.html) to automatically apply a set of resources. Note you must enable its feature gate before running `clusterctl init` (with `export EXP_CLUSTER_RESOURCE_SET=true`).
+
+## Guides
+
+- [Cluster API deployment - Docker provider (for local testing/development only)](docker/capi-docker.md)
+- [Cluster API deployment - Generic provider](generic/capi.md)
+
+## Testing
+
+You can test if the scheduler actually works by following this guide: 
+[Health Metric Example](https://github.com/intel/platform-aware-scheduling/blob/25a646ece15aaf4c549d8152c4ffbbfc61f8a009/telemetry-aware-scheduling/docs/health-metric-example.md)

telemetry-aware-scheduling/deploy/cluster-api/docker/capi-docker.md

@@ -0,0 +1,160 @@
+# Cluster API deployment - Docker provider (for local testing/development only)
+
+## Requirements
+
+- A management cluster provisioned in your infrastructure of choice and the relative tooling.
+  See [Cluster API Quickstart](https://cluster-api.sigs.k8s.io/user/quick-start.html).
+- Run Kubernetes v1.22 or greater (tested on Kubernetes v1.25).
+- Docker
+
+## Provision clusters with TAS installed using Cluster API
+
+We will provision a KinD cluster with the TAS installed using Cluster API. This guide is meant for local testing/development only.
+
+For the deployment using a generic provider, please refer to [Cluster API deployment - Generic provider](capi.md).
+
+1. Run the following to set up a KinD cluster for CAPD:
+
+```bash
+cat > kind-cluster-with-extramounts.yaml <<EOF
+kind: Cluster
+apiVersion: kind.x-k8s.io/v1alpha4
+nodes:
+- role: control-plane
+  extraMounts:
+    - hostPath: /var/run/docker.sock
+      containerPath: /var/run/docker.sock
+EOF
+```
+
+2. Enable the `CLUSTER_TOPOLOGY` feature gate:
+
+```bash
+export CLUSTER_TOPOLOGY=true
+```
+
+3. Initialize the management cluster:
+
+```bash
+clusterctl init --infrastructure docker
+```
+
+Run the following to generate the default cluster manifests:
+
+```bash
+clusterctl generate cluster capi-quickstart --flavor development \
+  --kubernetes-version v1.25.0 \
+  --control-plane-machine-count=3 \
+  --worker-machine-count=3 \
+  > capi-quickstart.yaml
+```
+
+Be aware that you will need to install a CNI such as Calico before the cluster will be usable. You may automate this
+step in the same way as we will see with TAS resources using ClusterResourceSets.
+
+2. Merge the contents of the resources provided in `cluster-patch.yaml`, `kubeadmcontrolplanetemplate-patch.yaml` and `clusterclass-patch.yaml` with
+   `your-manifests.yaml`.
+
+The new config will:
+- Configure TLS certificates for the extender
+- Change the `dnsPolicy` of the scheduler to `ClusterFirstWithHostNet`
+- Place `KubeSchedulerConfiguration` into control plane nodes and pass the relative CLI flag to the scheduler.
+- Change the behavior of the pre-existing patch application of `/spec/template/spec/kubeadmConfigSpec/files` in `ClusterClass` 
+such that our new patch is not ignored/overwritten. For some more clarification on this, see [this issue](https://github.com/kubernetes-sigs/cluster-api/pull/7630).
+
+You will also need to add a label to the `Cluster` resource of your new cluster to allow ClusterResourceSets to target
+it (see `cluster-patch.yaml`). Simply add a label `scheduler: tas` in your `Cluster` resource present in `your-manifests.yaml`.
+
+3. You will need to prepare the Helm Charts of the various components and join the TAS manifests together for convenience:
+
+First, under `telemetry-aware-scheduling/deploy/charts` tweak the charts if you need (e.g.
+additional metric scraping configurations), then render the charts:
+
+  ```bash
+  helm template ../charts/prometheus_node_exporter_helm_chart/ > prometheus-node-exporter.yaml
+  helm template ../charts/prometheus_helm_chart/ > prometheus.yaml
+  helm template ../charts/prometheus_custom_metrics_helm_chart > prometheus-custom-metrics.yaml
+  ```
+
+You need to add namespaces resources, else resource application will fail. Prepend the following to `prometheus.yaml`:
+
+  ```bash
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: monitoring
+  labels:
+    name: monitoring
+  ````
+
+Prepend the following to `prometheus-custom-metrics.yaml`:
+  ```bash
+kind: Namespace
+apiVersion: v1
+metadata:
+  name: custom-metrics
+  labels:
+    name: custom-metrics
+  ```
+
+The custom metrics adapter and the TAS deployment require TLS to be configured with a certificate and key.
+Information on how to generate correctly signed certs in kubernetes can be found [here](https://github.com/kubernetes-sigs/apiserver-builder-alpha/blob/master/docs/concepts/auth.md).
+Files ``serving-ca.crt`` and ``serving-ca.key`` should be in the current working directory.
+
+Run the following:
+
+  ```bash
+  kubectl -n custom-metrics create secret tls cm-adapter-serving-certs --cert=serving-ca.crt --key=serving-ca.key -oyaml --dry-run=client > custom-metrics-tls-secret.yaml
+  kubectl -n default create secret tls extender-secret --cert=serving-ca.crt --key=serving-ca.key -oyaml --dry-run=client > tas-tls-secret.yaml
+  ```
+
+**Attention: Don't commit the TLS certificate and private key to any Git repo as it is considered bad security practice! Make sure to wipe them off your workstation after applying the relative Secrets to your cluster.**
+
+You also need the TAS manifests (Deployment, Policy CRD and RBAC accounts) and the extender's "configmapgetter"
+ClusterRole. We will join the TAS manifests together, so we can have a single ConfigMap for convenience:
+
+  ```bash
+  yq '.' ../tas-*.yaml > tas.yaml
+  ```
+
+4. Create and apply the ConfigMaps
+
+  ```bash
+  kubectl create configmap custom-metrics-tls-secret-configmap --from-file=./custom-metrics-tls-secret.yaml -o yaml --dry-run=client > custom-metrics-tls-secret-configmap.yaml
+  kubectl create configmap custom-metrics-configmap --from-file=./prometheus-custom-metrics.yaml -o yaml --dry-run=client > custom-metrics-configmap.yaml
+  kubectl create configmap prometheus-configmap --from-file=./prometheus.yaml -o yaml --dry-run=client > prometheus-configmap.yaml
+  kubectl create configmap prometheus-node-exporter-configmap --from-file=./prometheus-node-exporter.yaml -o yaml --dry-run=client > prometheus-node-exporter-configmap.yaml
+  kubectl create configmap tas-configmap --from-file=./tas.yaml -o yaml --dry-run=client > tas-configmap.yaml
+  kubectl create configmap tas-tls-secret-configmap --from-file=./tas-tls-secret.yaml -o yaml --dry-run=client > tas-tls-secret-configmap.yaml
+  kubectl create configmap extender-configmap --from-file=../extender-configuration/configmap-getter.yaml -o yaml --dry-run=client > extender-configmap.yaml
+  ```
+
+Apply to the management cluster:
+
+  ```bash
+  kubectl apply -f '*-configmap.yaml'
+  ```
+
+5. Apply the ClusterResourceSets
+
+ClusterResourceSets resources are already given to you in `clusterresourcesets.yaml`.
+Apply them to the management cluster with `kubectl apply -f clusterresourcesets.yaml`
+
+6. Apply the cluster manifests
+
+Finally, you can apply your manifests `kubectl apply -f your-manifests.yaml`.
+The Telemetry Aware Scheduler will be running on your new cluster. You can connect to the workload cluster by
+exporting its kubeconfig:
+
+```bash
+clusterctl get kubeconfig ecoqube-dev > ecoqube-dev.kubeconfig
+```
+
+Then, specifically for the CAPD docker, point the kubeconfig to the correct address of the HAProxy container:
+
+```bash
+sed -i -e "s/server:.*/server: https:\/\/$(docker port ecoqube-dev-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g" ./ecoqube-dev.kubeconfig
+```
+
+You can test if the scheduler actually works by following this guide:
+[Health Metric Example](https://github.com/intel/platform-aware-scheduling/blob/25a646ece15aaf4c549d8152c4ffbbfc61f8a009/telemetry-aware-scheduling/docs/health-metric-example.md)

telemetry-aware-scheduling/deploy/cluster-api/docker/cluster-patch.yaml

@@ -0,0 +1,5 @@
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: Cluster
+metadata:
+  labels:
+    scheduler: tas

telemetry-aware-scheduling/deploy/cluster-api/docker/clusterclass-patch.yaml

@@ -0,0 +1,9 @@
+apiVersion: cluster.x-k8s.io/v1beta1
+kind: ClusterClass
+spec:
+  patches:
+    - definitions:
+        - jsonPatches:
+            - op: add
+              # Note: we must add a dash - after files, as shown below. Else the patch application in KubeadmControlPlaneTemplate will fail!
+              path: /spec/template/spec/kubeadmConfigSpec/files/-

telemetry-aware-scheduling/deploy/cluster-api/docker/clusterresourcesets.yaml

@@ -0,0 +1,83 @@
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: prometheus
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: prometheus-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: prometheus-node-exporter
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: prometheus-node-exporter-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: custom-metrics
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: custom-metrics-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: custom-metrics-tls-secret
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: custom-metrics-tls-secret-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: tas
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: tas-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: tas-tls-secret
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: tas-tls-secret-configmap
+---
+apiVersion: addons.cluster.x-k8s.io/v1alpha3
+kind: ClusterResourceSet
+metadata:
+  name: extender
+spec:
+  clusterSelector:
+    matchLabels:
+      scheduler: tas
+  resources:
+    - kind: ConfigMap
+      name: extender-configmap

telemetry-aware-scheduling/deploy/cluster-api/docker/kubeadmcontrolplanetemplate-patch.yaml

@@ -0,0 +1,56 @@
+apiVersion: controlplane.cluster.x-k8s.io/v1beta1
+kind: KubeadmControlPlaneTemplate
+spec:
+  template:
+    spec:
+      kubeadmConfigSpec:
+        clusterConfiguration:
+          scheduler:
+            extraArgs:
+              config: "/etc/kubernetes/schedulerconfig/scheduler-componentconfig.yaml"
+            extraVolumes:
+              - hostPath: "/etc/kubernetes/schedulerconfig"
+                mountPath: "/etc/kubernetes/schedulerconfig"
+                name: schedulerconfig
+              - hostPath: "/etc/kubernetes/pki/ca.key"
+                mountPath: "/host/certs/client.key"
+                name: cacert
+              - hostPath: "/etc/kubernetes/pki/ca.crt"
+                mountPath: "/host/certs/client.crt"
+                name: clientcert
+        initConfiguration:
+          patches:
+            directory: /etc/tas/patches
+        joinConfiguration:
+          patches:
+            directory: /etc/tas/patches
+        files:
+          - path: /etc/kubernetes/schedulerconfig/scheduler-componentconfig.yaml
+            content: |
+              apiVersion: kubescheduler.config.k8s.io/v1
+              kind: KubeSchedulerConfiguration
+              clientConnection:
+                kubeconfig: /etc/kubernetes/scheduler.conf
+              extenders:
+                - urlPrefix: "https://tas-service.default.svc.cluster.local:9001"
+                  prioritizeVerb: "scheduler/prioritize"
+                  filterVerb: "scheduler/filter"
+                  weight: 1
+                  enableHTTPS: true
+                  managedResources:
+                    - name: "telemetry/scheduling"
+                      ignoredByScheduler: true
+                  ignorable: true
+                  tlsConfig:
+                    insecure: false
+                    certFile: "/host/certs/client.crt"
+                    keyFile: "/host/certs/client.key"
+          - path: /etc/tas/patches/kube-scheduler+json.json
+            content: |-
+              [
+                  {
+                      "op": "add",
+                      "path": "/spec/dnsPolicy",
+                      "value": "ClusterFirstWithHostNet"
+                  }
+              ]