KernelCI K8S building clusters

[nuclearcat]

At current moment we have single k8s cluster with up to 40 nodes capacity. This is bare minimum, but since we migrate a lot of kernel and many new people are joining we need to expand and make k8s scheduling more smarter.

Future k8s job scheduling logic can be fairly simple:

• Pick random k8s cluster and check number of queued jobs that are not running yet (that indicates cluster is full), if it is 0 - schedule job there
• Pick next cluster and check number of queued jobs, if it is 0 - schedule job there, if we reached first cluster - based on number of queued jobs - schedule job on cluster with less queued jobs

Action items:

• Add more k8s clusters with inexpensive spot instances in different regions, but avoid to have egress costs between regions
• make k8s scheduling smarter
• forecast costs for each month based on current usage and planned growth