Upgrade platform Kubernetes to latest stable version

[rkalyanapurdue]

The production Kubernetes cluster on Jetstream2 is running a very old version of Kubernetes - v1.22. This needs to be upgraded ASAP.

[rkalyanapurdue]

This needs to happen soon since we are unable to add new nodes to the Kubernetes cluster as v1.22 cannot be installed from package repos any more.

[rkalyanapurdue]

Had a repeat of the NFS access issues (#11) following Jetstream2 outage, requiring new Kubernetes worker nodes to be created with the old v1.22

[rkalyanapurdue]

Tentatively plan for week of Jul 7

[rkalyanapurdue]

Steps:

1. Setup a Kubernetes cluster using the TerraForm/Ansible recipe
2. Make sure to retain current floating IP for the public JupyterHub since it is tied to the DNS entry
3. Modify NFS server exports to add the right fixed IPs for the new cluster nodes

[rkalyanapurdue]

We will do this after the Summer School since we don't want any disruptions then.

[rkalyanapurdue]

• Determine latest stable Kubernetes version
• Deploy a new Kubernetes cluster in the dev allocation using TF and Ansible
• For now, the MetalLB stuff will still be manual (including moving over the floating IP mapped to the domain)
• (optional) test integrating a GPU VM into the cluster and benchmark time taken