Merge tag 'vertical-pod-autoscaler-1.0.0' into vpa-1.0.0

Vertical Pod Autoscaler release 1.0.0
zalando-incubator · Mar 25, 2024 · 3ba3db4 · 3ba3db4
2 parents 4173828 + 276f1b8
commit 3ba3db4
Show file tree

Hide file tree

Showing 2,077 changed files with 345,388 additions and 51,666 deletions.
diff --git a/.gitignore b/.gitignore
@@ -28,3 +28,5 @@
 Session.vim
 .netrwhist
 
+# Binary files
+bin/
diff --git a/addon-resizer/enhancements/5700-nanny-configuration-reload/README.md b/addon-resizer/enhancements/5700-nanny-configuration-reload/README.md
@@ -0,0 +1,61 @@
+# KEP-5546: Automatic reload of nanny configuration when updated
+
+<!-- toc -->
+- [Summary](#summary)
+    - [Goals](#goals)
+    - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+    - [Notes](#notes)
+    - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+    - [Test Plan](#test-plan)
+<!-- /toc -->
+
+Sure, here's the enhancement proposal in the requested format:
+
+## Summary
+- **Goals:** The goal of this enhancement is to improve the user experience for applying nanny configuration changes in the addon-resizer 1.8 when used with the metrics server. The proposed solution involves automatically reloading the nanny configuration whenever changes occur, eliminating the need for manual intervention and sidecar containers. 
+- **Non-Goals:** This proposal does not aim to update the functional behavior of the addon-resizer.
+
+## Proposal
+The proposed solution involves updating the addon-resizer with the following steps:
+- Create a file system watcher using `fsnotify` under `utils/fswatcher` to watch nanny configurations' changes. It should run as a goroutine in the background.
+- Detect changes of the nanny configurations' file using the created `fswatcher` trigger the reloading process when configuration changes are detected. Events should be sent in a channel.
+- Re-execute the method responsible for building the NannyConfiguration `loadNannyConfiguration` to apply the updated configuration to the addon-resizer.
+- Proper error handling should be implemented to manage scenarios where the configuration file is temporarily inaccessible or if there are parsing errors in the configuration file.
+
+### Risks and Mitigations
+- There is a potential risk of filesystem-related issues causing the file watcher to malfunction. Proper testing and error handling should be implemented to handle such scenarios gracefully.
+- Errors in the configuration file could lead to unexpected behavior or crashes. The addon-resizer should handle parsing errors and fall back to the previous working configuration if necessary.
+
+## Design Details
+- Create a new package for the `fswatcher` under `utils/fswatcher`. It would contain the `fswatcher` struct and methods and unit-tests.
+    - `FsWatcher` struct would look similar to this:
+    ```go
+    type FsWatcher struct {
+        *fsnotify.Watcher
+
+        Events    chan struct{}
+        ratelimit time.Duration
+        names     []string
+        paths     map[string]struct{}
+    }
+    ```
+    - Implement the following functions:
+        - `CreateFsWatcher`: Instantiates a new `FsWatcher` and start watching on file system.
+        - `initWatcher`: Initializes the `fsnotify` watcher and initialize the `paths` that would be watched.
+        - `add`: Adds a new file to watch.
+        - `reset`: Re-initializes the `FsWatcher`.
+        - `watch`: watches for the configured files.
+- In the main function, we create a new `FsWatcher` and then we wait in an infinite loop to receive events indicating
+filesystem changes. Based on these changes, we re-execute `loadNannyConfiguration` function.
+
+> **Note:** The expected configuration file format is YAML. It has the same structure as the NannyConfiguration CRD.
+
+### Test Plan
+To ensure the proper functioning of the enhanced addon-resizer, the following test plan should be executed:
+1. **Unit Tests:** Write unit tests to validate the file watcher's functionality and ensure it triggers events when the configuration file changes.
+2. **Manual e2e Tests:** Deploy the addon-resizer with `BaseMemory` of `300Mi` and then we change the `BaseMemory` to `100Mi`. We should observer changes in the behavior of watched pod.
+
+
+[fsnotify]: https://github.com/fsnotify/fsnotify
diff --git a/balancer/deploy/controller.yaml b/balancer/deploy/controller.yaml
@@ -20,6 +20,12 @@ rules:
       - watch
       - patch
       - update
+  - apiGroups:
+      - balancer.x-k8s.io
+    resources:
+      - balancers/status
+    verbs:
+      - update
   - apiGroups:
       - ""
     resources:

diff --git a/builder/Dockerfile b/builder/Dockerfile
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.20
+FROM golang:1.20.4
 LABEL maintainer="Marcin Wielgus <[email protected]>"
 
 ENV GOPATH /gopath/

diff --git a/charts/cluster-autoscaler/Chart.yaml b/charts/cluster-autoscaler/Chart.yaml
@@ -11,4 +11,4 @@ name: cluster-autoscaler
 sources:
   - https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
 type: application
-version: 9.29.1
+version: 9.29.3
diff --git a/charts/cluster-autoscaler/README.md b/charts/cluster-autoscaler/README.md
@@ -411,6 +411,7 @@ vpa:
 | rbac.serviceAccount.name | string | `""` | The name of the ServiceAccount to use. If not set and create is `true`, a name is generated using the fullname template. |
 | replicaCount | int | `1` | Desired number of pods |
 | resources | object | `{}` | Pod resource requests and limits. |
+| secretKeyRefNameOverride | string | `""` | Overrides the name of the Secret to use when loading the secretKeyRef for AWS and Azure env variables |
 | securityContext | object | `{}` | [Security context for pod](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/) |
 | service.annotations | object | `{}` | Annotations to add to service |
 | service.create | bool | `true` | If `true`, a Service will be created. |

diff --git a/charts/cluster-autoscaler/templates/clusterrole.yaml b/charts/cluster-autoscaler/templates/clusterrole.yaml
@@ -151,15 +151,22 @@ rules:
     - cluster.x-k8s.io
     resources:
     - machinedeployments
-    - machinedeployments/scale
     - machinepools
-    - machinepools/scale
     - machines
     - machinesets
     verbs:
     - get
     - list
     - update
     - watch
+  - apiGroups:
+    - cluster.x-k8s.io
+    resources:
+    - machinedeployments/scale
+    - machinepools/scale
+    verbs:
+    - get
+    - patch
+    - update
 {{- end }}
 {{- end -}}
diff --git a/charts/cluster-autoscaler/templates/deployment.yaml b/charts/cluster-autoscaler/templates/deployment.yaml
@@ -80,7 +80,7 @@ spec:
             - --node-group-auto-discovery=mig:namePrefix={{ .name }},min={{ .minSize }},max={{ .maxSize }}
             {{- end }}
           {{- end }}
-          {{- if eq .Values.cloudProvider "oci-oke" }}
+          {{- if eq .Values.cloudProvider "oci" }}
             {{- if .Values.cloudConfigPath }}
             - --nodes={{ .minSize }}:{{ .maxSize }}:{{ .name }}
             - --balance-similar-node-groups
@@ -132,36 +132,36 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: AwsAccessKeyId
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
             {{- if .Values.awsSecretAccessKey }}
             - name: AWS_SECRET_ACCESS_KEY
               valueFrom:
                 secretKeyRef:
                   key: AwsSecretAccessKey
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
           {{- else if eq .Values.cloudProvider "azure" }}
             - name: ARM_SUBSCRIPTION_ID
               valueFrom:
                 secretKeyRef:
                   key: SubscriptionID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_RESOURCE_GROUP
               valueFrom:
                 secretKeyRef:
                   key: ResourceGroup
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_VM_TYPE
               valueFrom:
                 secretKeyRef:
                   key: VMType
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: AZURE_CLUSTER_NAME
               valueFrom:
                 secretKeyRef:
                   key: ClusterName
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- if .Values.azureUseWorkloadIdentityExtension }}
             - name: ARM_USE_WORKLOAD_IDENTITY_EXTENSION
               value: "true"
@@ -173,22 +173,22 @@ spec:
               valueFrom:
                 secretKeyRef:
                   key: TenantID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_CLIENT_ID
               valueFrom:
                 secretKeyRef:
                   key: ClientID
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: ARM_CLIENT_SECRET
               valueFrom:
                 secretKeyRef:
                   key: ClientSecret
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             - name: AZURE_NODE_RESOURCE_GROUP
               valueFrom:
                 secretKeyRef:
                   key: NodeResourceGroup
-                  name: {{ template "cluster-autoscaler.fullname" . }}
+                  name: {{ default (include "cluster-autoscaler.fullname" .) .Values.secretKeyRefNameOverride }}
             {{- end }}
           {{- end }}
           {{- range $key, $value := .Values.extraEnv }}

diff --git a/charts/cluster-autoscaler/templates/role.yaml b/charts/cluster-autoscaler/templates/role.yaml
@@ -49,16 +49,23 @@ rules:
     - cluster.x-k8s.io
     resources:
     - machinedeployments
-    - machinedeployments/scale
     - machinepools
-    - machinepools/scale
     - machines
     - machinesets
     verbs:
     - get
     - list
     - update
     - watch
+  - apiGroups:
+    - cluster.x-k8s.io
+    resources:
+    - machinedeployments/scale
+    - machinepools/scale
+    verbs:
+    - get
+    - patch
+    - update
 {{- end }}
 {{- if ( not .Values.rbac.clusterScoped ) }}
   - apiGroups:

diff --git a/charts/cluster-autoscaler/values.yaml b/charts/cluster-autoscaler/values.yaml
@@ -6,7 +6,7 @@ affinity: {}
 additionalLabels: {}
 
 autoDiscovery:
-  # cloudProviders `aws`, `gce`, `azure`, `magnum` and `clusterapi` `oci-oke` are supported by auto-discovery at this time
+  # cloudProviders `aws`, `gce`, `azure`, `magnum`, `clusterapi` and `oci` are supported by auto-discovery at this time
   # AWS: Set tags as described in https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup
 
   # autoDiscovery.clusterName -- Enable autodiscovery for `cloudProvider=aws`, for groups matching `autoDiscovery.tags`.
@@ -396,3 +396,6 @@ vpa:
   updateMode: "Auto"
   # vpa.containerPolicy -- [ContainerResourcePolicy](https://github.com/kubernetes/autoscaler/blob/vertical-pod-autoscaler/v0.13.0/vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go#L159). The containerName is always et to the deployment's container name. This value is required if VPA is enabled.
   containerPolicy: {}
+
+# secretKeyRefNameOverride -- Overrides the name of the Secret to use when loading the secretKeyRef for AWS and Azure env variables
+secretKeyRefNameOverride: ""
diff --git a/cluster-autoscaler/.gitignore b/cluster-autoscaler/.gitignore
@@ -1,6 +1,7 @@
 cluster-autoscaler
 cluster-autoscaler-amd64
 cluster-autoscaler-arm64
+cluster-autoscaler-s390x
 cluster_autoscaler
 .cover
 

diff --git a/cluster-autoscaler/Dockerfile.s390x b/cluster-autoscaler/Dockerfile.s390x
@@ -0,0 +1,20 @@
+# Copyright 2016 The Kubernetes Authors. All rights reserved
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+ARG BASEIMAGE=gcr.io/distroless/static:nonroot-s390x
+FROM $BASEIMAGE
+LABEL maintainer="Marcin Wielgus <[email protected]>"
+
+COPY cluster-autoscaler-s390x /cluster-autoscaler
+WORKDIR /
+CMD ["/cluster-autoscaler"]
diff --git a/cluster-autoscaler/FAQ.md b/cluster-autoscaler/FAQ.md
@@ -29,6 +29,7 @@ this document:
   * [I'm running cluster with nodes in multiple zones for HA purposes. Is that supported by Cluster Autoscaler?](#im-running-cluster-with-nodes-in-multiple-zones-for-ha-purposes-is-that-supported-by-cluster-autoscaler)
   * [How can I monitor Cluster Autoscaler?](#how-can-i-monitor-cluster-autoscaler)
   * [How can I increase the information that the CA is logging?](#how-can-i-increase-the-information-that-the-ca-is-logging)
+  * [How can I change the log format that the CA outputs?](#how-can-i-change-the-log-format-that-the-ca-outputs)
   * [How can I see all the events from Cluster Autoscaler?](#how-can-i-see-all-events-from-cluster-autoscaler)
   * [How can I scale my cluster to just 1 node?](#how-can-i-scale-my-cluster-to-just-1-node)
   * [How can I scale a node group to 0?](#how-can-i-scale-a-node-group-to-0)
@@ -125,8 +126,8 @@ Since version 1.0.0 we consider CA as GA. It means that:
 
  * We have enough confidence that it does what it is expected to do. Each commit goes through a big suite of unit tests
    with more than 75% coverage (on average). We have a series of e2e tests that validate that CA works well on
-   [GCE](https://k8s-testgrid.appspot.com/sig-autoscaling#gce-autoscaling)
-   and [GKE](https://k8s-testgrid.appspot.com/sig-autoscaling#gke-autoscaling).
+   [GCE](https://testgrid.k8s.io/sig-autoscaling#gce-autoscaling)
+   and [GKE](https://testgrid.k8s.io/sig-autoscaling#gke-autoscaling).
    Due to the missing testing infrastructure, AWS (or any other cloud provider) compatibility
    tests are not the part of the standard development or release procedure.
    However there is a number of AWS users who run CA in their production environment and submit new code, patches and bug reports.
@@ -862,7 +863,7 @@ Events:
 ```
 
 This limitation was solved with
-[volume topological scheduling](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/volume-topology-scheduling.md)
+[volume topological scheduling](https://github.com/kubernetes/design-proposals-archive/blob/main/storage/volume-topology-scheduling.md)
 introduced as beta in Kubernetes 1.11 and planned for GA in 1.13.
 To allow CA to take advantage of topological scheduling, use separate node groups per zone.
 This way CA knows exactly which node group will create nodes in the required zone rather than relying on the cloud provider choosing a zone for a new node in a multi-zone node group.
@@ -923,6 +924,20 @@ or infrastructure endpoints, then setting a value of `--v=9` will show all the i
 HTTP calls made. Be aware that using verbosity levels higher than `--v=1` will generate
 an increased amount of logs, prepare your deployments and storage accordingly.
 
+### How Can I change the log format that the CA outputs?
+
+There are 2 log format options, `text` and `json`. By default (`text`), the Cluster Autoscaler will output 
+logs in the [klog native format](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog-output).
+```
+I0823 17:15:11.472183   29944 main.go:569] Cluster Autoscaler 1.28.0-beta.0
+```
+
+Alternatively, adding the flag `--logging-format=json` changes the 
+[log output to json](https://kubernetes.io/docs/concepts/cluster-administration/system-logs/#klog-output).
+```
+{"ts":1692825334994.433,"caller":"cluster-autoscaler/main.go:569","msg":"Cluster Autoscaler 1.28.0-beta.0\n","v":1}
+```
+
 ### What events are emitted by CA?
 
 Whenever Cluster Autoscaler adds or removes nodes it will create events

diff --git a/cluster-autoscaler/Makefile b/cluster-autoscaler/Makefile
@@ -1,4 +1,4 @@
-ALL_ARCH = amd64 arm64
+ALL_ARCH = amd64 arm64 s390x
 all: $(addprefix build-arch-,$(ALL_ARCH))
 
 TAG?=dev
@@ -109,3 +109,23 @@ test-in-docker: clean docker-builder
 	docker run ${RM_FLAG} -v `pwd`:/cluster-autoscaler/:Z autoscaling-builder:latest bash -c 'cd /cluster-autoscaler && go test -race ./... ${TAGS_FLAG}'
 
 .PHONY: all build test-unit clean format execute-release dev-release docker-builder build-in-docker release generate push-image push-manifest
+
+## Location to install dependencies to
+LOCALBIN ?= $(shell pwd)/bin
+$(LOCALBIN):
+	mkdir -p $(LOCALBIN)
+
+## Tool Binaries
+CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
+
+## Tool Versions
+CONTROLLER_TOOLS_VERSION ?= v0.13.0
+
+.PHONY: controller-gen
+controller-gen: $(CONTROLLER_GEN) ## Download controller-gen locally if necessary.
+$(CONTROLLER_GEN): $(LOCALBIN)
+	test -s $(LOCALBIN)/controller-gen || GOBIN=$(LOCALBIN) go install sigs.k8s.io/controller-tools/cmd/controller-gen@$(CONTROLLER_TOOLS_VERSION)
+
+.PHONY: manifest
+manifest: controller-gen ## Generate WebhookConfiguration, ClusterRole and CustomResourceDefinition objects.
+	$(CONTROLLER_GEN) rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd
diff --git a/cluster-autoscaler/OWNERS b/cluster-autoscaler/OWNERS
@@ -1,4 +1,5 @@
 approvers:
+- BigDarkClown
 - feiskyer
 - towca
 - x13n

diff --git a/cluster-autoscaler/README.md b/cluster-autoscaler/README.md
@@ -48,6 +48,7 @@ Starting from Kubernetes 1.12, versioning scheme was changed to match Kubernetes
 
 | Kubernetes Version  | CA Version   |
 |--------|--------|
+| 1.28.X | 1.28.X |
 | 1.27.X | 1.27.X |
 | 1.26.X | 1.26.X |
 | 1.25.X | 1.25.X |
@@ -86,10 +87,11 @@ target ETA and the actual releases.
 | Date       | Maintainer Preparing Release | Backup Maintainer |
 |------------|------------------------------|-------------------|
 | 2023-03-15 | MaciekPytel                  | gjtempleton       |
-| 2023-05-17 | gjtempleton                  | towca             |
-| 2023-07-19 | towca                        | x13n              |
-| 2023-09-13 | x13n                         | MaciekPytel       |
-| 2023-11-15 | MaciekPytel                  | gjtempleton       |
+| 2023-05-17 | gjtempleton                  | BigDarkClown      |
+| 2023-07-19 | BigDarkClown                 | towca             |
+| 2023-09-13 | towca                        | x13n              |
+| 2023-11-15 | x13n                         | MaciekPytel       |
+| 2024-01-17 | MaciekPytel                  | gjtempleton       |
 
 Additional patch releases may happen outside of the schedule in case of critical
 bugs or vulnerabilities.