infra.ci/release.ci jobs are failing with pod not ready

[halkeye]

Service

• infra.ci.jenkins.io
• release.ci.jenkins.io

Summary

Since the 1.31.x line of the kubernetes-plugin, our controllers using AKS for pod agents have their build failing 100% with the following error:

io.fabric8.kubernetes.client.KubernetesClientException: not ready after 5000 MILLISECONDS
	at io.fabric8.kubernetes.client.utils.Utils.waitUntilReadyOrFail(Utils.java:176)
	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:322)
	at io.fabric8.kubernetes.client.dsl.internal.core.v1.PodOperationsImpl.exec(PodOperationsImpl.java:84)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:427)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:344)
	at hudson.Launcher$ProcStarter.start(Launcher.java:507)
	at org.jenkinsci.plugins.durabletask.PowershellScript.doLaunch(PowershellScript.java:176)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launchWithCookie(FileMonitoringTask.java:139)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:132)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:324)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at jdk.internal.reflect.GeneratedMethodAccessor44.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:158)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
	at WorkflowScript.run(WorkflowScript:273)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at jdk.internal.reflect.GeneratedMethodAccessor183.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:402)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:314)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:278)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)

There is NO error if the pod definition has only a single container named jnlp because there is no kubectl exec call (the pipeline steps are executed in the container named jnlp through the agent.jar process, and not by any call to the Kubernetes API.

This issue is already reported by some JIRA issues:

• https://issues.jenkins.io/browse/JENKINS-67474
• https://issues.jenkins.io/browse/JENKINS-59652?focusedCommentId=414369&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-414369
• https://issues.jenkins.io/browse/JENKINS-64848

Reproduction steps

• Release.ci.jenkins.io: https://release.ci.jenkins.io/job/core/job/package/job/master/177/console

[dduportal]

https://support.cloudbees.com/hc/en-us/articles/360054642231-Considerations-for-Kubernetes-Clients-Connections-when-using-Kubernetes-Plugin describes the 4 different leverages that we can use for now:

• The 2 leverages related to increasing timeout + allowing more parallel connections to Kubernetes API have already been exhausted: 10000 allowed connections + 5 min timeout (which is HUGE) are not solving the issue and not changing anything
• "merging" sh/container/other" pipelien steps to leverage the amount connections to Kubernetes API are not working: even with a freshly restarted controller with
• Short term for infra and release: we take the "all in one container" road: each of our usual containers are going to switch their base image to the jenkinsci/inbound-agent:jdk11 one (or jenkinsci/inbound-agent:alpine-jdk11).

[dduportal]

• docker-builder changes: Extend from inbound-agent docker-builder#30
• docker-helmfile changes: fix: use jnlp inbound-agent image as source, bump aws-cli version and add yamllint docker-helmfile#45
• kubernetes-management changes: fix: use unique complete image to avoid calling kubectl exec and getting (tcp) errors kubernetes-management#1881
• Pipeline Library change (for the docker image standard builds + updatecli): https://github.com/jenkins-infra/pipeline-library/pull/273/files

[dduportal]

• Note: Switch to debian base caused Docker build with img failing on infra.ci with SYNC_USERMAP_ACK: got 255: Invalid argument #2190 for docker-builder

[dduportal]

For releases:

• fix: switch linux packaging to single-container JNLP with dduportal image release#197
• Bump package image to jenkinsciinfra/packaging:2.0.0 release#198
• Release & Package Processes: "one container to rule them all" release#201

[dduportal]

• Note: Windows package image for the release to do

[dduportal]

• The images jenkinsciinfra/terraform (https://github.com/jenkins-infra/docker-terraform) and jenkinsciinfra/packer (https://github.com/jenkins-infra/docker-packer) were merged in a single new one: jenkinsciinfra/hashicorp-tools (https://github.com/jenkins-infra/docker-hashicorp-tools), and their source repositories were archived.
  This new image is inheriting from JNLP as well.

[dduportal]

Closing at it seems that we covered all images.