Support spot-to-spot consolidation.

[nikmohan123]

Description
This PR is a feature for enabling spot-to-spot "replace" consolidation for AWS Karpenter. It provides an option for the customers to enable/disable the spot-to-spot consolidation through a feature flag named as SpotToSpotConsolidation. By default the feature is turned off. The spot consolidation would kick-in only when there are a minimum of 15 instanceTypes for the new NodeClaim to replace the current spot candidate. This condition is valid for single-node spot-to-spot consolidation. The value (15) for minimum number of InstanceTypes is currently hard coded based on the analysis performed on the flexibility of AWS customer requests.

How was this change tested?
End to end test passed.

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: # TODO.
• [] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod ready!

Name	Link
🔨 Latest commit	89ff489
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/65a1c0baaf8d5600084da246
😎 Deploy Preview	https://deploy-preview-5377--karpenter-docs-prod.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[coveralls]

Pull Request Test Coverage Report for Build 7499032831

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.02%) to 82.53%

---

Totals
Change from base Build 7496532085:	-0.02%
Covered Lines:	4951
Relevant Lines:	5999

---

💛 - Coveralls

[njtran]

Can you update the go.mod to bring in your fork's changes? That way we can start testing the tests you've included here.

[njtran]

/karpenter snapshot

[github-actions]

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:v0-93edef38e8b1474e030625dd1a8926f85d911a09.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

[jonathan-innis]

/karpenter snapshot

[github-actions]

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:v0-fcf4b09c07e23e299072db697dcb4cf898f11ad4.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

[jonathan-innis]

/karpenter snapshot

[github-actions]

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:v0-88fc7461f438e7fc182bed7133f8c3c4eed1f42c.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

[jonathan-innis]

/karpenter snapshot

[jonathan-innis]

/karpenter snapshot

[github-actions]

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:v0-b15b38122eb3351eea749b7daf05033373613ce9.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

[jonathan-innis]

LGTM 🚀

[jonathan-innis]

/karpenter snapshot

[github-actions]

Snapshot successfully published to oci://021119463062.dkr.ecr.us-east-1.amazonaws.com/karpenter/snapshot/karpenter:v0-fb142e61e17b05fa15dcec872008cba15094aa51.
To install you must login to the ECR repo with an AWS account:

aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 021119463062.dkr.ecr.us-east-1.amazonaws.com

[jonathan-innis]

LGTM 🚀