kubeadm init phase for apiserver does not accept multi valued flag in extra-args

[supershal]

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:36:44Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:
Hardware

• Kubernetes version (use kubectl version):

• Cloud provider or hardware configuration:
  Hardware: 1x1 cluster. 8 cpu, 16 GB memory

• OS (e.g. from /etc/os-release):
  NAME="CentOS Linux"
  VERSION="7 (Core)"

• Kernel (e.g. uname -a):
  3.10.0-693.2.2.el7.x86_64

What happened?

executing kubeadm init phase control-plane apiserver --apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists" throwing following error.

 error: invalid argument "enable-admission-plugins=LimitRanger,NamespaceExists" for "--apiserver-extra-args" flag: malformed pair, expect string=string

apiserver-extra-args expects map[string]string which will be comma separated nameX=valueX pairs.
however the enable-admission-plugins is a multi valued flag for api-server. so kubeadm errors out if the value of the flag is multi valued as shown in example above.

What you expected to happen?

kubeadm init phase control-plane apiserver flag --apiserver-extra-args should be able to accept extra args with multi valued flag.

How to reproduce it (as minimally and precisely as possible)?

Fail: execute with multi valued flag. kubeadm init phase control-plane apiserver --apiserver-extra-args "enable-admission-plugins=LimitRanger,NamespaceExists"

Pass: execute with single valued flag. kubeadm init phase control-plane apiserver --apiserver-extra-args "enable-admission-plugins=LimitRanger"

Anything else we need to know?

After little bit exploring the code:
The flag --apiserver-extra-args is instance of MapStringString (https://github.com/kubernetes/apiserver/blob/master/pkg/util/flag/map_string_string.go#L30) with NoSplit=false which will not accept multi valued flag.
The flag initialization: https://github.com/kubernetes/kubernetes/blob/release-1.13/cmd/kubeadm/app/cmd/options/generic.go#L49

Please also suggest if there is any work around until this issue is resolved.

[neolit123]

this is a bug.
thanks for the report @supershal
https://github.com/kubernetes/kubernetes/blob/a22763b24ee53b5efca5c60b000a63875a6b6f2a/staging/src/k8s.io/component-base/cli/flag/map_string_string.go#L25-L29

we need to set NoSplit=false

in the meantime the workaround and actual recommended method to handle the extra arguments is to use the kubeadm configuration file:
https://kubernetes.io/docs/setup/independent/control-plane-flags/#apiserver-flags
https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

[RA489]

@neolit123 can i take this issue?

[rosti]

@RA489 is working on this
/lifecycle active

[rosti]

we need to set NoSplit=false

@neolit123 Actually we need to set NoSplit = true (currently it's false). In that way we can use it on command line like the user suggested. Unfortunately, this has the potential to break some users as stuff that was previously invoked as --apiserver-extra-args "Arg1=Val1,Arg2=Val2" now have to be invoked as --apiserver-extra-args "Arg1=Val1" --apiserver-extra-args "Arg2=Val2".

We can deprecate this flag and, either make the users use the config file, or introduce a new NoSplit flag.

Also, what we do for this flag has to be done for the --controller-manager-extra-args and --scheduler-extra-args as they work in the same way.

[RA489]

@rosti IIUC if we want Multiple flag invocations like --apiserver-extra-args "Arg1=Val1" --apiserver-extra-args "Arg2=Val2" then we need to set 'NoSplit=true' and when it is invoked as comma-separated key-value pairs
like --apiserver-extra-args "Arg1=Val1,Arg2=Val2" then we need to set 'NoSplit=false'.

[rosti]

@RA489 precisely. In this issue the user cannot specify an API server parameter that can take multiple comma separated values (that would be --enable-admission-plugins=LimitRanger,NamespaceExists for the API Server in our case).
So the current way of having NoSplit=false does not allow us to do that via the command line.

[neolit123]

@rosti i see what you mean.

https://github.com/kubernetes/kubernetes/blob/a22763b24ee53b5efca5c60b000a63875a6b6f2a/staging/src/k8s.io/component-base/cli/flag/map_string_string.go#L25-L34

with both true and false it would not work.

this:

sudo kubeadm init phase control-plane apiserver --apiserver-extra-args
"enable-admission-plugins=SomePlugin" --apiserver-extra-args "enable-admission-plugins=LimitRanger"

sets the value for plugins to LimitRanger only

MapStringString is not suitable for enable-admission-plugins.
so we either need a new flag or document that you cannot use this on the CLI.

i vote for the later.

[neolit123]

/assign
/kind documetation

i will write a quick note about this in the Troubleshoot docs and move this item outside of 1.14 in terms of code changes.

[neolit123]

sent this docs PR:
kubernetes/website#12937

might as well close this eventually as adding new custom flag for this is not ideal, but for now punting to next cycle.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/lifecycle frozen

[neolit123]

merging this issue with:
#1601

both config and CLI flags are not handled correctly in this regard.