Upgrade failed when using patches directory

[kaviankarimzadeh]

What keywords did you search in kubeadm issues before filing this one?

patches

BUG REPORT

Versions

Current: 1.28.4
Upgrade to : 1.29.8

NAME="AlmaLinux"
VERSION="8.10 (Cerulean Leopard)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.10 (Cerulean Leopard)"

uname -a
Linux kubm01 4.18.0-553.el8_10.x86_64 x86_64 x86_64 x86_64 GNU/Linux

containerd://1.6.31

What happened?

We have some args already added to our kube-system StaticPods like etcd, kube-apiserver that needs to be persist during upgrade. until now we were passing --config flag with a path to a file include ClusterConfiguration and all configs that must be persist. But some of these args like encryption-provider-config in apiserver or listen-metrics-urls in etcd gets removed when we run upgrade, so we need to manually add/update those values during upgrade process so the upgrade can continue.
And also using --config flags is not recommended for upgrade and soon it'll be deprecated.

so we decided to use --patches flags, but it seems none of the patchStrategy (merge, strategic, json) are capable of adding args to the current args already exist in those StaticPods.

here's an example with merge patchStrategy:

file name: etcd0+merge.yaml / etcd0+strategic.yam

apiVersion: v1
kind: Pod
metadata:
  name: etcd
spec:
  containers:
  - name: etcd
    command:
      - /usr/local/bin/etcd
    args:
      - --quota-backend-bytes=8589934592
      - --listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381

here's an example with json patchStrategy :

file name: etcd0+json.json

[
   {
       "op": "add",
       "path": "/spec/containers/0/command/-",
       "value":  [ "--quota-backend-bytes=8589934592" ]
   },
   {
       "op": "add",
       "path": "/spec/containers/0/command/-",
       "value":  [ "--listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381" ]
   }
]

Upgrade command:

kubeadm upgrade apply v1.29.8  --patches /etc/kubernetes/patches/ --dry-run

No args added to to the StaticPod yaml file.

What you expected to happen?

We're expecting to add additional args to kube-system StaticPods during upgrade process while keeping what is already there in their specs. ( not overriding )

How to reproduce it (as minimally and precisely as possible)?

here's an example with merge patchStrategy:

file name: /etc/kubernetes/patches/etcd0+merge.yaml

apiVersion: v1
kind: Pod
metadata:
  name: etcd
spec:
  containers:
  - name: etcd
    command:
      - /usr/local/bin/etcd
    args:
      - --quota-backend-bytes=8589934592
      - --listen-metrics-urls=http://127.0.0.1:2381,https://192.68.210.21:2381

kubeadm upgrade apply v1.29.8  --patches /etc/kubernetes/patches/ --dry-run

Anything else we need to know?

Here in this document an example provided to check available patchStrategy for PodSpec:

https://v1-29.docs.kubernetes.io/docs/tasks/manage-kubernetes-objects/update-api-object-kubectl-patch/#notes-on-the-strategic-merge-patch

"io.k8s.api.core.v1.PodSpec": {
    ...,
    "containers": {
        "description": "List of containers belonging to the pod.  ...."
    },
    "x-kubernetes-patch-merge-key": "name",
    "x-kubernetes-patch-strategy": "merge"
}

But hew in OpemApi spec for args there's no patchStrategy provided:
https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json

    "io.k8s.api.core.v1.Container": {
      "description": "A single application container that you want to run within a pod.",
      "properties": {
        "args": {
          "description": "Arguments to the entrypoint. The container image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. \"$$(VAR_NAME)\" will produce the string literal \"$(VAR_NAME)\". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell",
          "items": {
            "type": "string"
          },
          "type": "array",
          "x-kubernetes-list-type": "atomic"
        }

[neolit123]

we have an upgrade e2e test that makes sure this works.

you should try the following:

$ cat ./patches/etcd0+json.json
[
   {
       "op": "add",
       "path": "/spec/containers/0/command/-",
       "value":  "--quota-backend-bytes=8589934592"
   },
]

note there are not [] around the flag value
add more flags in separate "add" objects.

results in:

$ sudo cat /etc/kubernetes/manifests/etcd.yaml | grep 858
    - --quota-backend-bytes=8589934592

/kind support