Upgrade to Kubernetes 1.30

[dduportal]

Per https://learn.microsoft.com/en-us/azure/aks/supported-kubernetes-versions?tabs=azure-cli#aks-kubernetes-release-calendar, Microsoft stops supporting Kubernetes 1.29 in March 2025.

• Note: EKS has the same support expiration policy for 1.29 (March 2025) - Upgrade to Kubernetes 1.30 #4258

As such, we must upgrade our

AZURE: 1.30.7

https://kubernetes.io/blog/2024/04/17/kubernetes-v1-30-release/

AWS: 1.30.9

aws eks describe-cluster-versions --region us-east-2 --profile=terraform-admin | jq '.clusterVersions[] | select(.clusterVersion == "1.30")'

{
    "clusterVersion": "1.30",
    "clusterType": "eks",
    "defaultPlatformVersion": "eks.27",
    "defaultVersion": false,
    "releaseDate": "2024-05-23T02:00:00+02:00",
    "endOfStandardSupportDate": "2025-07-23T02:00:00+02:00",
    "endOfExtendedSupportDate": "2026-07-23T02:00:00+02:00",
    "status": "STANDARD_SUPPORT",
    "kubernetesPatchVersion": "1.30.9"
}

Last issue for 1.29 was: https://github.com//issues/4161

---

Task list:

• Upgrade kubectl within packer-images All-in-One

  • First upgrade the updatecli manifest - chore(updatecli/kubectl) switch to 1.30 kubectl cli version packer-images#1733
  • Then apply the updatecli PR that is produced by this change: - Bump kubectl version to 1.30.10 packer-images#1737
  • Trigger a release of the packer-images - https://github.com/jenkins-infra/packer-images/releases/tag/2.28.1
  • Then apply the Kubernetes-Management PR that will use this new version - Bump agent templates for infra.ci.jenkins.io (packer-image 2.28.1) kubernetes-management#6231
  • Also apply the puppet (jenkins-infra/jenkins-infra) - Bump Packer Agent Templates (All-In-One) Version to 2.28.1 jenkins-infra#3891

• Check AKS changelog for 1.30 - https://github.com/Azure/AKS/blob/master/CHANGELOG.md and AWS changelog - https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions-standard.html#kubernetes-1-30

• Upgrade AKS cluster cijenkinsio-agents-1 (details) planned today Tuesday 18 February

• Upgrade AWS cluster cijenkinsio-agents-2 (details) planned Wednesday 19 February

• Upgrade AKS cluster infracijio-agents-1 (details) planned Wednesday 19 February

• Upgrade AKS cluster privatek8s(details) planned Thursday 20 of February morning paris time

• Upgrade AKS cluster publick8s (details) planned Thursday 20 of February afternoon paris time

• Add Logo ;)

• Open issue for next Kubernetes version

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 92% Upgrade to Kubernetes 1.31 #4546
2. 83% Upgrade to Kubernetes 1.28 #4144
3. 83% Upgrade to Kubernetes 1.27 #3948
4. 83% Upgrade to Kubernetes 1.26 #3683
5. 83% Upgrade to Kubernetes 1.25 #3582
6. 83% Upgrade to Kubernetes 1.24 #3387
7. 83% Upgrade to Kubernetes 1.23 #3053

[dduportal]

Update: delayed triage until October 2024

[smerle33]

packer-image release was stuck and relaunched https://infra.ci.jenkins.io/job/infra-tools/job/packer-images/view/tags/job/2.28.1/

[smerle33]

infracijio-agents-1

• Announcement: jenkins-infra element channel
• Disable kubernetes management for infracijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - feat(pipeline): remove infracijioagents1 for kub 1.30 upgrade kubernetes-management#6252

Warning

Terraform upgrade must be performed manually otherwise it will upgrade the cluster where agents are currently performing, leading to build failure

• Upgrade cluster to 1.30 with Terraform (Azure)
  • PR with checks all green: feat(infracijioagents1): upgrade to kubernetes 1.30 azure#938
  • Prepare local (admin laptop) terraform plan
  • Disable Terraform Azure Job in infra.ci
  • Merge the PR
  • ⚠️ Note that AKS version is not always exactly the same patch as kubectl . Always use the AKS changelog to check (or the az CLI). here should be 1.30.7
  • Deploy from local admin laptop
    ⚠️ hotfix jenkins-infra/azure@77dc7e8 wrong variable
  • Enable Terraform Azure Job in infra.ci and run it once with success - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1488/ Apply complete! Resources: 1 added, 0 changed, 0 destroyed. for reports.jenkins.io
• Enable kubernetes management for infracijio-agents-1 cluster and check it runs successfully - Revert "feat(pipeline): remove infracijioagents1 for kub 1.30 upgrade" kubernetes-management#6253

---

[smerle33]

Changelog

Check of the Kubernetes 1.30 and AKS changelogs:

Nothing for us:
- deprecation https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#deprecation
- not used: Proper casing will be enforced on PUT of Microsoft.ContainerService/managedClusters/agentPools for the AgentPoolMode property. See this Azure/AKS#4468 for more detail.

Notes:
- NOTE: Azure portal mentions we're using a deprecated resource (ComponentStatus). Culprit is the datadog helm chart (see DataDog/helm-charts#961) but it's should still work for 1.30 as it's still in the master branch source code https://github.com/liggitt/kubernetes/blob/4e7e14203db8cde906604b057b1b2a8a15e8a50d/pkg/apis/core/types.go#L6377

[smerle33]

Update: publick8s cluster upgrade:

Warning

Your cluster was found to be using the deprecated In-Tree Persistent Volume (PV) storage class.
one PV for ldap-backup using azureFile
need to upgrade with https://learn.microsoft.com/en-us/azure/aks/csi-migrate-in-tree-volumes

done with jenkins-infra/azure#940

• Announcement:
  • Open datetime announcement on status.jenkins.io - open issue publick8s upgrade kubernetes 1.30 status#597
  • developer mailing list (Authentication outage due to LDAP restart) -groups.google.com message link
  • IRC #jenkins-infra
• Disable kubernetes management for publick8s cluster (remove it from Jenkinsfile_k8s) - feat(pipeline): remove publick8s for kub 1.30 upgrade kubernetes-management#6262
• Terraform PR to upgrade cluster - feat(publick8s): upgrade to kubernetes 1.30 azure#942
• Merge Terraform PR and let the upgrade happen
• Check the LDAP restart successfully once upgrade is performed. If the pod restarted and is running, try the service by logging-in on ci.jenkins.io and repo.jenkins-ci.org (checks that pods are all running)
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1496/. - Resources: 0 added, 0 changed, 0 destroyed.
• Enable kubernetes management and revert exclusion of publick8s cluster, checking both diff and apply run successfully - Revert "feat(pipeline): remove publick8s for kub 1.30 upgrade" kubernetes-management#6267
• Check the pagerduty alerts are all gone
• Close announcements

[smerle33]

cijioagents1 not used anymore so easy remnants of azure ci ... will be remove soon

• Announcements: not needed
• Disable kubernetes management for cijio-agents-1 cluster (remove it from Jenkinsfile_k8s) - feat(cijioagents1) removed from the jenkinsfile for kub 1.30 upgrade kubernetes-management#6243
• Terraform PR to upgrade cluster to 1.30.7 but NO merge - feat(cijioagents1) upgrade to kubernetes 1.30.7 azure#937
  • Verify all checks are ok
  • Merge the PR
  • Verify upgraded is performed with no error
• Enable kubernetes management for ci.jio-agents-1 cluster and check it runs successfully - https://github.com/jenkins-infra/kubernetes-management/pull/id we keep ci.jio-agents-1 out of the kubernetes management to prepare it's removing next week.

[smerle33]

cijioagents2 on AWS

We plan to handle this upgrade Wednesday 19 February

• Announcements:
  • status.jenkins.io - open issue cijioagents2 upgrade kubernetes 1.30 status#592
  • developer mailing list - https://groups.google.com/g/jenkinsci-dev/c/hjNHy_VsGgQ
  • IRC #jenkins-infra -
  • Add a banner on ci.jenkins.io -
• Disable kubernetes management for cijio-agents-2 cluster (remove it from Jenkinsfile_k8s) - feat(cijioagents2) upgrade to kubernetes 1.30.9 kubernetes-management#6250
• Disable the 2 clouds defined in ci.jenkins.io on puppet (jenkins-infra/jenkins-infra) for the operation ci.jenkins.io-agents-1-bom and ci.jenkins.io-agents-2 - feat(aws.ci.jenkins.io): disable 2 kub clouds for kub 1.30 upgrade jenkins-infra#3903
• Terraform PR to upgrade cluster to 1.29.7 but NO merge - https://github.com/jenkins-infra/azure/pull/
  • Verify all checks are ok
  • Merge the PR
  • Verify upgraded is performed with no error
• Enable kubernetes management for ci.jio-agents-2 cluster and check it runs successfully - feat(pipeline): re enable cijioagents2 after kub 1.30 upgrade kubernetes-management#6251
• Re-Enable the 2 ci.jio puppet clouds and check container agents are scheduled with the Infra/acceptance-test job - feat(aws.ci.jenkins.io): re-enable 2 kub clouds for kub 1.30 upgrade jenkins-infra#3904 and acceptance test: https://ci.jenkins.io/job/Infra/job/acceptance-tests/job/check-agent-availability/4678/pipeline-graph/
• Close Annoucements
  • status.jenkins.io - close cijioagents2 kub 1.30 upgrade status#594
  • developer mailing list + jenkins-infra mailing
  • IRC #jenkins-infra
  • Remove the banner on ci.jenkins.io

[smerle33]

privatek8s on AZURE cluster upgrade:

• Announcement:
  • Open datetime announcement on status.jenkins.io - open issue privatek8s upgrade kubernetes 1.30 status#595
  • IRC #jenkins-infra
• Disable kubernetes management for privatek8s cluster (remove it from Jenkinsfile_k8s) - feat(pipeline): remove privatek8s for kub 1.30 upgrade kubernetes-management#6254
• Terraform PR to upgrade cluster - feat(privatek8s): upgrade to kubernetes 1.30 azure#939
• Merge Terraform PR and let the upgrade happen
• Check terraform job and eventually trigger it for success with no changes - https://infra.ci.jenkins.io/job/terraform-jobs/job/azure/job/main/1492/
• Enable kubernetes management and revert exclusion of privatek8s cluster, checking both diff and apply run successfully - Revert "feat(pipeline): remove privatek8s for kub 1.30 upgrade" kubernetes-management#6260
• Check for services healthiness (pagerduty, datadog, manual control, logging in on infra.ci.jenkins.io/release.ci.jenkins.io with the Infra Agents Health job https://release.ci.jenkins.io/job/Infra%20Agents%20Health/) - https://release.ci.jenkins.io/job/Infra%20Agents%20Health/37/
• Close announcements

[dduportal]

cijioagents2 on AWS

The upgrade made us learn and realize a few improvement/fixes are needed in the Terraform AWS project, related to version tracking of EKS version and its components:

• [updatecli] retrieve EKS version when tracking AMI node version terraform-aws-sponsorship#141
• [updatecli] Allow passing flag --profile to all the manifests terraform-aws-sponsorship#142
• EKS: Kubernetes upgrade should include AMI and addons to ensure full compatibility terraform-aws-sponsorship#143
• [updatecli] Track the Karpenter Linux Node AMIs versions terraform-aws-sponsorship#144

It is important to note that:

• These improvements/fixes should be done before the Kubernetes 1.31
• These changes would be useful so we can finely track version of AKS and its components with the same technique

[smerle33]