[WIP] Support for Kubernetes v1.32.0 🚧

docs/development/new-kubernetes-version.md

@@ -51,16 +51,27 @@ There is a CI/CD job that runs periodically and releases a new `hyperkube` image
 
 ### Adapting Gardener
 
+<!-- // TODO(marc1404): Create issue for fixing the `compare-k8s-feature-gates.sh` script -->
+
 - Allow instantiation of a Kubernetes client for the new minor version and update the `README.md`:
   - See [this](https://github.com/gardener/gardener/pull/5255/commits/63bdae022f1cb1c9cbd1cd49b557545dca2ec32a) example commit.
   - The list of supported versions is meanwhile maintained [here](../../pkg/utils/validation/kubernetesversion/version.go) in the `SupportedVersions` variable.
 - Maintain the Kubernetes feature gates used for validation of `Shoot` resources:
   - The feature gates are maintained in [this](../../pkg/utils/validation/features/featuregates.go) file.
-  - To maintain this list for new Kubernetes versions, run `hack/compare-k8s-feature-gates.sh <old-version> <new-version>` (e.g. `hack/compare-k8s-feature-gates.sh v1.26 v1.27`).
-  - It will present 3 lists of feature gates: those added and those removed in `<new-version>` compared to `<old-version>` and feature gates that got locked to default in `<new-version>`.
-  - Add all added feature gates to the map with `<new-version>` as `AddedInVersion` and no `RemovedInVersion`.
-  - For any removed feature gates, add `<new-version>` as `RemovedInVersion` to the already existing feature gate in the map.
-  - For feature gates locked to default, add `<new-version>` as `LockedToDefaultInVersion` to the already existing feature gate in the map.
+  - To maintain this list for new Kubernetes versions follow this guide:
+    - **Alpha & Beta Feature Gates:**
+      - Open: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features
+      - Search the page for the new Kubernetes version, e.g. "1.32".
+      - Add new alpha feature gates that have been added "Since" the new Kubernetes version.
+      - Change the `Default` for Beta feature gates that have been promoted "Since" the new Kubernetes version.
+    - **Graduated & Deprecated Feature Gates:**
+      - Open: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-graduated-or-deprecated-features
+      - Search the page for the new Kubernetes version, e.g. "1.32".
+      - Change `LockedToDefaultInVersion` for GA and Deprecated feature gates that have been graduated/deprecated "Since" the new Kubernetes version.
+    - **Removed Feature Gates:**
+      - Open: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/#feature-gates-that-are-removed
+      - Search the page for the **current** Kubernetes version, e.g. if the new version is "1.32", search for "1.31".
+      - Set `RemovedInVersion` to the **new** Kubernetes version for feature gates that have been removed after the **current** Kubernetes version according to the "To" column. 
   - See [this](https://github.com/gardener/gardener/pull/5255/commits/97923b0604300ff805def8eae981ed388d5e4a83) example commit.
 - Maintain the Kubernetes `kube-apiserver` admission plugins used for validation of `Shoot` resources:
   - The admission plugins are maintained in [this](../../pkg/utils/validation/admissionplugins/admissionplugins.go) file.

docs/usage/shoot-operations/supported_k8s_versions.md

@@ -16,7 +16,7 @@ The minimum version of a seed cluster that can be connected to Gardener is **`1.
 
 ## Shoot Clusters
 
-Gardener itself is capable of spinning up clusters with Kubernetes versions **`1.25`** up to **`1.31`**.
+Gardener itself is capable of spinning up clusters with Kubernetes versions **`1.25`** up to **`1.32`**.
 However, the concrete versions that can be used for shoot clusters depend on the installed provider extension.
 Consequently, please consult the documentation of your provider extension to see which Kubernetes versions are supported for shoot clusters.
 

example/gardener-local/kind/cluster/values.yaml

@@ -1,4 +1,4 @@
-image: kindest/node:v1.31.2@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e
+image: kindest/node:v1.32.0@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027
 
 gardener:
   apiserverRelay:

example/operator/20-garden.yaml

@@ -83,7 +83,7 @@ spec:
           capacity: 10Gi
         # className: default
     kubernetes:
-      version: 1.31.1
+      version: 1.32.0
     # kubeAPIServer:
     #   eventTTL: 1h
     #   featureGates:

example/provider-local/garden/cloudprofile/cloudprofile.yaml

@@ -8,6 +8,7 @@ spec:
   - name: local
   kubernetes:
     versions:
+    - version: 1.32.0
     - version: 1.31.1
     - version: 1.30.0
     - version: 1.29.0

pkg/provider-local/node/Dockerfile

@@ -1,4 +1,4 @@
-FROM kindest/node:v1.31.2@sha256:18fbefc20a7113353c7b75b5c869d7145a6abd6269154825872dc59c1329912e
+FROM kindest/node:v1.32.0@sha256:c48c62eac5da28cdadcf560d1d8616cfa6783b58f0d94cf63ad1bf49600cb027
 
 ARG TARGETOS
 ARG TARGETARCH

pkg/resourcemanager/controller/node/criticalcomponents/helper/daemon_controller.go

@@ -46,7 +46,7 @@ import (
 //     Returns true when a daemonset should continue running on a node if a daemonset pod is already
 //     running on that node.
 //
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/controller/daemon/daemon_controller.go#L1275-L1330
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/controller/daemon/daemon_controller.go#L1275-L1306
 func NodeShouldRunDaemonPod(node *corev1.Node, ds *appsv1.DaemonSet) (bool, bool) {
 	pod := NewPod(ds, node.Name)
 
@@ -73,7 +73,7 @@ func NodeShouldRunDaemonPod(node *corev1.Node, ds *appsv1.DaemonSet) (bool, bool
 }
 
 // predicates checks if a DaemonSet's pod can run on a node.
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/controller/daemon/daemon_controller.go#L1275-L1330
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/controller/daemon/daemon_controller.go#L1308-L1318
 func predicates(pod *corev1.Pod, node *corev1.Node, taints []corev1.Taint) (fitsNodeName, fitsNodeAffinity, fitsTaints bool) {
 	fitsNodeName = len(pod.Spec.NodeName) == 0 || pod.Spec.NodeName == node.Name
 	// Ignore parsing errors for backwards compatibility.
@@ -86,7 +86,7 @@ func predicates(pod *corev1.Pod, node *corev1.Node, taints []corev1.Taint) (fits
 }
 
 // NewPod creates a new pod
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/controller/daemon/daemon_controller.go#L1275-L1330
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/controller/daemon/daemon_controller.go#L1320-L1330
 func NewPod(ds *appsv1.DaemonSet, nodeName string) *corev1.Pod {
 	newPod := &corev1.Pod{Spec: ds.Spec.Template.Spec, ObjectMeta: ds.Spec.Template.ObjectMeta}
 	newPod.Namespace = ds.Namespace
@@ -99,7 +99,7 @@ func NewPod(ds *appsv1.DaemonSet, nodeName string) *corev1.Pod {
 }
 
 // AddOrUpdateDaemonPodTolerations apply necessary tolerations to DaemonSet Pods, e.g. node.kubernetes.io/not-ready:NoExecute.
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/controller/daemon/util/daemonset_util.go#L47-L102
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/controller/daemon/util/daemonset_util.go#L47-L102
 func AddOrUpdateDaemonPodTolerations(spec *corev1.PodSpec) {
 	// DaemonSet pods shouldn't be deleted by NodeController in case of node problems.
 	// Add infinite toleration for taint notReady:NoExecute here
@@ -158,7 +158,7 @@ func AddOrUpdateDaemonPodTolerations(spec *corev1.PodSpec) {
 
 // AddOrUpdateTolerationInPodSpec tries to add a toleration to the toleration list in PodSpec.
 // Returns true if something was updated, false otherwise.
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/apis/core/v1/helper/helpers.go#L261-L287
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/apis/core/v1/helper/helpers.go#L261-L287
 func AddOrUpdateTolerationInPodSpec(spec *corev1.PodSpec, toleration *corev1.Toleration) bool {
 	podTolerations := spec.Tolerations
 
@@ -187,7 +187,7 @@ func AddOrUpdateTolerationInPodSpec(spec *corev1.PodSpec, toleration *corev1.Tol
 
 // Semantic can do semantic deep equality checks for core objects.
 // Example: apiequality.Semantic.DeepEqual(aPod, aPodWithNonNilButEmptyMaps) == true
-// Copied from https://github.com/kubernetes/kubernetes/blob/v1.31.0/pkg/apis/core/helper/helpers.go#L92-L114
+// Copied from https://github.com/kubernetes/kubernetes/blob/v1.32.0/pkg/apis/core/helper/helpers.go#L92-L114
 var Semantic = conversion.EqualitiesOrDie(
 	func(a, b resource.Quantity) bool {
 		// Ignore formatting, only care that numeric value stayed the same.

pkg/utils/kubernetes/controllers.go

@@ -85,45 +85,47 @@ var APIGroupControllerMap = map[string]map[string]versionutils.VersionRange{
 		"storage-version-migrator-controller": {AddedInVersion: "1.30"},
 	},
 	"v1": {
-		"attachdetach":                         {},
-		"bootstrapsigner":                      {},
-		"cloud-node":                           {},
-		"cloud-node-lifecycle":                 {},
-		"cronjob":                              {},
-		"csrapproving":                         {},
-		"csrsigning":                           {},
-		"daemonset":                            {},
-		"deployment":                           {},
-		"disruption":                           {},
-		"endpoint":                             {},
-		"endpointslice":                        {},
-		"endpointslicemirroring":               {},
-		"ephemeral-volume":                     {},
-		"garbagecollector":                     {RemovedInVersion: "1.30"},
-		"horizontalpodautoscaling":             {},
-		"job":                                  {},
-		"legacy-service-account-token-cleaner": {AddedInVersion: "1.28"},
-		"namespace":                            {},
-		"nodelifecycle":                        {},
-		"persistentvolume-binder":              {},
-		"persistentvolume-expander":            {},
-		"podgc":                                {},
-		"pv-protection":                        {},
-		"pvc-protection":                       {},
-		"replicaset":                           {},
-		"replicationcontroller":                {},
-		"resource-claim-controller":            {AddedInVersion: "1.27"},
-		"resourcequota":                        {},
-		"root-ca-cert-publisher":               {},
-		"route":                                {},
-		"service":                              {},
-		"service-cidr-controller":              {AddedInVersion: "1.29"},
-		"serviceaccount":                       {},
-		"serviceaccount-token":                 {},
-		"statefulset":                          {},
-		"taint-eviction-controller":            {AddedInVersion: "1.29"},
-		"tokencleaner":                         {},
-		"ttl":                                  {},
-		"ttl-after-finished":                   {},
+		"attachdetach":             {},
+		"bootstrapsigner":          {},
+		"cloud-node":               {},
+		"cloud-node-lifecycle":     {},
+		"cronjob":                  {},
+		"csrapproving":             {},
+		"csrsigning":               {},
+		"daemonset":                {},
+		"deployment":               {},
+		"disruption":               {},
+		"endpoint":                 {},
+		"endpointslice":            {},
+		"endpointslicemirroring":   {},
+		"ephemeral-volume":         {},
+		"garbagecollector":         {RemovedInVersion: "1.30"},
+		"horizontalpodautoscaling": {},
+		"job":                      {},
+		"kube-apiserver-serving-clustertrustbundle-publisher": {AddedInVersion: "1.32"},
+		"legacy-service-account-token-cleaner":                {AddedInVersion: "1.28"},
+		"namespace":                                           {},
+		"nodelifecycle":                                       {},
+		"persistentvolume-binder":                             {},
+		"persistentvolume-expander":                           {},
+		"podgc":                                               {},
+		"pv-protection":                                       {},
+		"pvc-protection":                                      {},
+		"replicaset":                                          {},
+		"replicationcontroller":                               {},
+		"resource-claim-controller":                           {AddedInVersion: "1.27"},
+		"resourcequota":                                       {},
+		"root-ca-cert-publisher":                              {},
+		"route":                                               {RemovedInVersion: "1.32"},
+		"service":                                             {RemovedInVersion: "1.32"},
+		"service-cidr-controller":                             {AddedInVersion: "1.29"},
+		"serviceaccount":                                      {},
+		"serviceaccount-token":                                {},
+		"statefulset":                                         {},
+		"taint-eviction-controller":                           {AddedInVersion: "1.29"},
+		"tokencleaner":                                        {},
+		"ttl":                                                 {},
+		"ttl-after-finished":                                  {},
+		"volumeattributesclass-protection":                    {AddedInVersion: "1.32"},
 	},
 }

pkg/utils/validation/admissionplugins/admissionplugins.go

@@ -49,6 +49,7 @@ var (
 		"ImagePolicyWebhook":                   {},
 		"LimitPodHardAntiAffinityTopology":     {},
 		"LimitRanger":                          {},
+		"MutatingAdmissionPolicy":              {VersionRange: versionutils.VersionRange{AddedInVersion: "1.32"}},
 		"MutatingAdmissionWebhook":             {Required: true},
 		"NamespaceAutoProvision":               {},
 		"NamespaceExists":                      {},