[INFRA-3100] Migrate updates.jenkins.io to another Cloud

[jenkins-infra-bot]

Why

Read the EPIC (aws cost decrease: #2646)

What

• updates.jenkins.io serves the JSON of the update center, but it is not behind a CDN (need to be updated regularly) causing a lot of outbound data transfer (2.4 Mb for a JSON download, esitmation around 300 Gb of data sent daily, estimation around 10 Tb per month which costs some $$$.

• The VM pkg.origin.jenkins.io is managing multiple services and separting them could be a great help.

How

Different paths can be taken here: to be discussed in infra meeting + validated by the board as it's an important service.

• Migrating the service to Oracle Cloud:
  • the first 10 Tb of outbound data is free, and if the worst case, 50 Tb of outbound would cost less than 3k $ per month - https://www.oracle.com/be/cloud/networking/networking-pricing.html
  • ARM machines: cheaper and better performance for a simple webserver + SSH + data
  • Block storage: close to archives.jenkins.io
  • Risk: Oracle sponsoring program is only 1 year, and is not easily secured for paiment as other account

• Migrating to Azure
  • Easily in AKS: easier management
  • Safer paiment process
  • Risk: cost of outbound to be evaluated

---

Originally reported by dduportal, imported from: Migrate updates.ci.jenkins.io to another Cloud

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[note]

• Check that Claim that updates.jenkins.io certificate is sometimes expiring #3017 is not happening once migration is finished.
• The error curl: (16) Error in the HTTP2 framing layer should not happen anymore with the latest Apache versions

[dduportal]

Requires setting up the Oracle Terraform project, like #2682

Todo list:

• Specify a new infra (VM + data storage of the same size as the actual pkg.origin.jenkins.io machine + any other Oracle infra requirements). Implies checking the pricing to get an overall idea
• Once VM is created, add it to puppet management: new node, with the role / profiles associated to "update.jenkins.io"
• Rsync the data one time
• Update the jenkins-infra/update-center2 's associated job in trusted.ci to update both "pkg.origin.jenkins.io" VM and the new one (⚠️ inline pipeline, don't search for a Jenkinsfile as code)
• Validate the new instance (including checking with Daniel, Tim and other contributors)
• Communicate about the upcoming change
• As proposed by Stephane, use a round-robin DNS record value to split traffic between old/new during some time and check for error

[dduportal]

Blocked by #2973

[smerle33]

actual machine size is :
32Gb RAM
8cpu
1,2Tb data disk (372Gb free)

about half of the power is used currently (checked with the local SAR probe)

[smerle33]

Infra to specify :

• 1 VM (similar to archive.jenkins.io)
• 1 network ("mirrors") + 1 subnet ("20210630-1531")
• 1 volume 1,2Tb
• 1 set of security groups to restrict network access
• 1 ssh key pair

[smerle33]

VM specifications :

• 4vCPU/16Gb RAM : half of actual machine
• proposal to use ARM like archive.jenkins.io --> 4ocpu (VM Type Standard A1 Flex)
• Image Ubuntu 20.04 (FULL [non minimal]) --> upgrade from ubuntu 18.04 for actual machine
  • option for 22.04 but need testing for puppet agent (+ openssl v3)

[dduportal]

Update:

• HTTP/404 errors on tool installer metadata were reported in 404 downloading some tool installer JSONs (geo dependant) #4403. Problem has been fixed

[dduportal]

Update:

• [Update Center] "eventually consistent" Azure Shared File storage for HTTPD htdocs (aka. corrupted files some time to time) #4402 (SMB + azcopy unreliable data storage has been done (switch to NFS + rsync) and looks like a good long term solution.
• There has been a major outage before the switch to NFS with the SMB CSI mount stealing all the VM CPUs leading to major issues. It looks like it was also caused by an Azure Storage account latency increase (1s to 2s per req !!). Incident closed and should not happen (in the way how NFS CSI works).
  • Wrong update-center returned for 2.452.4 #4427
  • HTTP errors on multiple sites #4428
• Opened [Update Center] HTTP/404 on /current/updates/*.json* links #4432 (minor)
• We still need to work on [Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas #4401 has we had Cloudflare outages today