Add GPU right sizing for GPU slices created by instaslice and apply the recommendations

[bharathappali]

Describe the feature

Kruize currently works along side with VPA to apply the recommendations, but VPA doesn't apply the GPU partitioning changes to apply the accelerator recommendations generated by kruize. Instaslice Operator on Openshift actually partitions the GPU on the fly when ever the pod resources for GPU are changed. So If kruize can track the MIG partition usage and come up with MIG right sizing recommendations, kruize can apply them via instaslice (As on when kruize changes the pod requests and limits wrt to GPU instaslice can create the slice and assign to the pod). This way the accelerator recommendations can also be applied via kruize.

Suggest a solution

Kruize should be able to track the MIG partition (which are created by instaslice) usage and come up with accelerator recommendations and apply them by updating the kubernetes deployment / statefulset / job object with latest requests and limits and let instaslice create the GPU slices on the fly.

Checklist of changes to be done

• Add required permissions to kruize to update/patch the kubernetes objects
• Add required permissions to kruize to read the instaslice object
• Make changes to the metrics recording logic to incorporate the metrics to track the MIG's created by instaslice
• Make changes to the recommendation logic to consider the MIG usage
• Create a custom resource updater for updating resources via kruize