[ci.jenkins.io] Move ACI agents to ephemeral Windows containers to AWS

[dduportal]

Requires #4319

Goal: stop running Windows container agents in ACI and use Kubernetes instead

• Need to add Windows Node Pool(s?) in the EKS cluster - https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html
  • Need to use Windows 2019 for now as we use container image built on top of nanoserver 1809: https://github.com/jenkins-infra/docker-inbound-agents/blob/17c545c52b7dfb1f329fd7a02e53f1a899b27f7b/maven/jdk17/Dockerfile.nanoserver#L24
  • Taints are needed (like https://github.com/jenkins-infra/azure/blob/e81b9697e4a9cedebf10e119b6a5a112e09b651f/privatek8s.tf#L227-L230)
• Need to add pod template with tolerations
  • Like https://github.com/jenkins-infra/release/blob/52dbc24d709a4f91328b2a2dcafe2db6f006a2c9/PodTemplates.d/package-windows.yaml#L47-L55

(edit) Given we chose to use EKS with Karpenter, the following tasks are required, following what is described in https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html

• The permissions eks:kube-proxy-windows is required in the role assigned to nodes
  • Ref. https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html#enable-windows-support (point 5.)
• The VPC CNI plugin must be set up to support Windows IPAM (otherwise we'll have weird errors when trying to start pods, related to sandbox creation found and missing pod tags with VPC IPv4)
  • Ref. https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html#enable-windows-support (point 3. and 4.)
  • Terraform configuration of the addon: https://aws-ia.github.io/terraform-aws-eks-blueprints/snippets/vpc-cni-custom-networking/
  • Helm chart of the VPC-CNI add on: https://github.com/aws/eks-charts/blob/master/stable/aws-vpc-cni/README.md
• Karpenter Node Pools:
  • We need both 2019 and 2022
  • We need to adapt the disk setup to avoid habving a 20Gb C: and an unused 300Gb D:
• Implement JCasc setup for Windows Kubernetes containers
• Add the 4 definitions for Windows container agents (maven-8-window/maven-windows, maven-11-windows, maven-17-windows and maven-21-windows)
• (Optimization) Given the size of the container images, we should to set up an ECR pull through cache (using custom naming as we control pods) to decrease agent startup time

[dduportal]

Adding this issue to the milestone:

• Triggered a bit earlier than expected to help on [Incident] Windows build of plugins don't start on ci.jenkins.io #4490
• Will use Windows Node Pool from Karpenter as per [ci.jenkins.io] Create private EKS cluster with "side" services (datadog, ACP, etc.) #4319 (comment)
  • Note: the following page might be usefull: Windows Pods not resolving DNS names aws/karpenter-provider-aws#5099 (comment)

[dduportal]

Update: The initial EKS requirements have been implemented with success. A single Pod extracted from a local Jenkins controller with the upcoming JCasc setup definition has been tested:

apiVersion: v1
kind: Pod
metadata:
  labels:
    jenkins/label: test-ddu
  name: jnlp-maven-21-windows-test-ddu
  namespace: jenkins-agents
spec:
  containers:
  - command:
    - pwsh.exe
    - -f
    - C:/ProgramData/Jenkins/jenkins-agent.ps1
    env:
    - name: PATH
      value: C:/tools/jdk-21/bin;$PATH
    - name: JENKINS_SECRET
      value: SuperSecret
    - name: ARTIFACT_CACHING_PROXY_SERVERID
      value: http://127.0.0.1:8080/
    - name: JAVA_HOME
      value: C:/tools/jdk-21
    - name: JENKINS_DIRECT_CONNECTION
      value: ci.jenkins.io:50000
    - name: JENKINS_AGENT_WORKDIR
      value: C:/Jenkins/agent
    - name: JENKINS_JAVA_OPTS
      value: -XX:+PrintCommandLineFlags
    - name: JENKINS_JAVA_BIN
      value: C:/tools/jdk-17/bin/java
    - name: JENKINS_PROTOCOLS
      value: JNLP4-connect
    - name: JENKINS_AGENT_NAME
      value: jnlp-maven-21-windows-test-ddu
    - name: TEMP
      value: C:/Windows/Temp
    - name: TMP
      value: C:/Windows/Temp
    - name: REMOTING_OPTS
      value: -noReconnectAfter 1d
    - name: JENKINS_INSTANCE_IDENTITY
      value: Secret
    - name: JENKINS_NAME
      value: jnlp-maven-21-windows-2llr2
    image: jenkinsciinfra/inbound-agent-maven:jdk21-nanoserver@sha256:302f2003cc52011cac1b9769b983dfd1490a520d8cdca7b0519c9f26e07c3802
    # image: jenkins/inbound-agent:3283.v92c105e0f819-8-jdk17-nanoserver-ltsc2022
    imagePullPolicy: IfNotPresent
    name: jnlp
    resources:
      limits:
        cpu: "2"
        memory: 4G
      requests:
        cpu: "2"
        memory: 4G
    volumeMounts:
    - mountPath: C:/Users/Administrator/.m2/repository
      name: volume-1
    - mountPath: C:/Windows/Temp
      name: volume-0
    - mountPath: C:/Jenkins/agent
      name: workspace-volume
    workingDir: C:/Jenkins/agent
  nodeSelector:
    kubernetes.io/arch: amd64
    kubernetes.io/os: windows
  restartPolicy: Never
  serviceAccount: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: ci.jenkins.io/agents
    operator: Equal
    value: "true"
  - effect: NoSchedule
    key: ci.jenkins.io/windows-2019
    operator: Equal
    value: "true"
  volumes:
  - emptyDir:
      medium: Memory
    name: volume-0
  - emptyDir: {}
    name: volume-1
  - emptyDir: {}
    name: workspace-volume

[dduportal]

Update: Real life tests in progress with the first set of pod templates are set up with "test" labels as they need a bit more testing.

• Initial test shows pods not scheduled because there was no node pools with the proper taints and labels. Forgot to set up the "ci.jenkins.io/agents" toleration.

  • Fix tested manually and as code in fix(ci.jenkins.io) add missing toleration and agentJavaBin for windows pod templates jenkins-infra#3900

• Then, pods are scheduled but are failing with the error below. Scheduling (both node and pod) is good, and the default command (e.g. container entrypoint) is good, but there is an inbound agent script setup error:

  Start-Process: C:\ProgramData\Jenkins\jenkins-agent.ps1:162
  Line |
   162 |      Start-Process -FilePath $JAVA_BIN -Wait -NoNewWindow -ArgumentLis …
       |      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       | This command cannot be run due to the error: The system cannot find the
       | file specified.
  

  • See https://github.com/jenkinsci/docker-agent/blob/4aed4a7e0d68c35d4ac558bfe39e74a1f1fe6dd9/jenkins-agent.ps1#L162
  • Root cause: the default Agent Java bin to use (C:/openjdk-17/bin/java) is different than the Windows VMs (C:/tools/jdk-17/bin/java), because, currently, our images are built on top of the inbound agent which provides the agent JDK, while we install our custom JDKs per declination (8, 11, 17, 21) which is different than the all in one with all JDKs.
  • Fix tested manually and as code in fix(ci.jenkins.io) add missing toleration and agentJavaBin for windows pod templates jenkins-infra#3900

• With the Agent Java Bin fixed, new startup error:

  Unrecognized VM option ''
  Error: Could not create the Java Virtual Machine.
  Error: A fatal exception has occurred. Program will exit.
  

  • Need to check the value passed to JENKINS_JAVA_OPTS. Currently the pod specifies:

    - name: JENKINS_JAVA_OPTS
      value: -XX:+PrintCommandLineFlags

  • Manual "hack" is to remove the env var and continue (to make sure all errors are caught). Gotta work on the real fix.

• Then, with the correct Agent Java bin and without any JENKINS_JAVA_OPTS env. var, we have the following error:

  INFO: Connecting to ci.jenkins.io:50000
  Feb 17, 2025 6:06:39 PM hudson.remoting.Launcher$CuiListener error
  SEVERE: null
  java.nio.channels.UnresolvedAddressException
          at java.base/sun.nio.ch.Net.checkAddress(Net.java:149)
          at java.base/sun.nio.ch.Net.checkAddress(Net.java:157)
          at java.base/sun.nio.ch.SocketChannelImpl.checkRemote(SocketChannelImpl.java:816)
          at java.base/sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:839)
          at java.base/java.nio.channels.SocketChannel.open(SocketChannel.java:285)
          at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:231)
          at hudson.remoting.Engine.connectTcp(Engine.java:1131)
          at hudson.remoting.Engine.innerRun(Engine.java:999)
          at hudson.remoting.Engine.run(Engine.java:586)
  

  • This is a DNS resolution error. It's weird because the instance profile shows the eks:kube-proxy-windows IAM permissions (required for DNS resolution) is present. We'll have to diagnose this further

[dduportal]

Update:

DNS resolution error.

This one was tricky. Fixed in jenkins-infra/terraform-aws-sponsorship#138.
TL;DR;

• https://docs.aws.amazon.com/eks/latest/userguide/windows-support.html#enable-windows-support specifies (points 4. and 5.) that the eks:kube-proxy-windows permission is required (so the Windows Service kube-proxy sets up the DNS and routes properly internally)
• I mistook this permission for an IAM permission in fix(ci.jio-agents-2) add missing IAM permissions to EKS Node IAM role to support Windows nodes terraform-aws-sponsorship#133
• The solution was simple yet not really intuitive due to the presence of Karpenter. As per Access Entry for Windows managed node groups terraform-aws-modules/terraform-aws-eks#2994, Karpenter terraform module is not expecting additional node roles: there is only one (which also has permissions on the SQS queue used by Karpenter), which makes it hard to segregate Linux and Windows pools. But setting the module's access entry to type EC2_WINDOWS is good enough as it is a subset of EC2_LINUX (e.g. no regression for Linux nodes).

Next steps:

• Fix the "JENKINS_JAVA_OPTS" issue (see comment above)
• Set up the PATH for Windows containers in Puppet
• Validate the existing 4 templates
• Start using ECR for the 4 templates and validate them again

Future optimizations (might need another issue) from AWS:

• Build custom Windows AMIs to enable Fast Launch (as it is REALLY efficient for AWS EC2 VM agents): https://docs.aws.amazon.com/eks/latest/best-practices/windows-ami.html#_configuring_faster_launching_for_custom_eks_optimized_amis
• Cache the jenkins inbound agent layer (and eventually temurin) on the custom AMIs as per https://docs.aws.amazon.com/eks/latest/best-practices/windows-ami.html#_caching_windows_base_layers_on_custom_amis

[dduportal]

Update:

• The JENKINS_JAVA_OPTS fix has been fixed in fix(jenkinscontroller/kubernetes-clouds) ensure Windows container have quoted JENKINS_JAVA_OPTS jenkins-infra#3908 (missing quote in the case of Windows)
• Successful test in https://ci.jenkins.io/job/Infra/job/acceptance-tests/job/check-agent-availability/4687: each agent type shows the correct java version both with mvn -v and java -version.

[dduportal]

Real life test in progress with the Remoting build:

• https://ci.jenkins.io/job/Core/job/remoting/job/master/792 is a manual replay forced to only build on Windows (JDK17) with the platform set to the test agent label maven-17-windows-test.
  • Node is already up with the JDK17 container image already pulled: agent allocation was fast (of course).

=> lets compare with previous build 791 which tests were in timeout around 15 min after the build start

[dduportal]

Real life test in progress with the Remoting build:

* https://ci.jenkins.io/job/Core/job/remoting/job/master/792 is a manual replay forced to only build on Windows (JDK17) with the platform set to the test agent label `maven-17-windows-test`.
  
  * Node is already up with the JDK17 container image already pulled: agent allocation was fast (of course).

=> lets compare with previous build 791 which tests were in timeout around 15 min after the build start

Test was successful! Let's roll!

[dduportal]

Update:

• Windows containers are back on ci.jenkins 🥳
• Container Images are cached in ECR, which should bring (a bit) faster agent startup
  • BUT: [ci.jenkins.io] Windows EKS container agents #4548

Tested with success one last time on both remoting and infra/acceptance tests