diff --git a/designs/odcr.md b/designs/odcr.md index 82b18eac6434..a2b96abf6e28 100644 --- a/designs/odcr.md +++ b/designs/odcr.md @@ -275,22 +275,38 @@ Pros: Cons: - ***TBD*** -#### Pricing and consolidation +## Drift -##### Provisioning +[TODO: Add a section on Drift] + +## Pricing/Consolidation + +### Provisioning Pricing is directly considered during provisioning and consolidation as capacity-reservation is prepaid. It is assumed to have a price of 0 during provisioning. -##### Consolidating Capacity Reserved Instances +### Consolidation During consolidation pricing does matter as it affects which candidate will be [prioritized](https://github.com/kubernetes-sigs/karpenter/blob/75826eb51589e546fffb594bfefa91f3850e6c82/pkg/controllers/disruption/consolidation.go#L156). Since all capacity instances are paid ahead of time, their cost is already incurred. Users would likely want to prioritize filling their reserved capacity. -reservation first then fall back into other instances. Because of this reserved instances should likely show up as 0 dollar pricing when we calculate the instance pricing. Since each candidate is tied to a NodePool and EC2NodeClass, we should be able to safely override the pricing per node under a capacity reservation. +reservation first then fall back into other instances. Because of this reserved instances should likely show up as 0 dollar pricing when we calculate the instance pricing. -##### Consolidating into Capacity Reserved Instances +#### Consolidating into Capacity Reserved Instances -If we track Capacity Reservation usage, we can optimize the cluster configuration by moving non-Capacity Reserved instances into +If we track Capacity Reservation usage, we can optimize the cluster configuration by moving non-Capacity Reserved instances into Capacity Reserved instances. We would need to match the instance type, platform and availability zone prior to doing this. +This would be done by the standard consolidation algorithm and should work with minimal changes, since consolidation already optimizes for cost. + +#### Consolidating between Capacity Reservations + +Treating the price of all capacity reservation offerings as `0` sounds sensical but has some adverse interactions with consolidation. Most notably, if we launch an instance into a capacity reservation offering, we will _never_ consolidate out of that offering -- even if there is a smaller instance type that would work just as well for the pods on that node. + +In practicality, that larger capacity reservation could have been used for other work -- perhaps on another cluster, but may have been held by a single pod that prevented Karpenter from consolidating it. + +To solve for this edge case, we won't model the pricing of capacity reservations as `0` but as a "near-0" value. We'll divide the existing price of the on-demand offering by 1000 to represent a price that is less than every other instance type offering, but still maintains the relative ordering of on-demand instance types. + +In practice, this means that if a user has two capacity reservation offerings available: one for a `c6a.48xlarge` and another for a `c6a.large`, where we launch into the `c6a.48xlarge` first, we will still be able to consolidate down to the `c6a.large` when pods are scaled back down. + #### Labels When a node is launched against a CapacityReservation we will expose Capacity Reservation @@ -315,7 +331,7 @@ The main failure scenario is when Capacity Reservation limit is hit and no new n 1. We filter inside Karpenter before calling CreateFleet API and throwing an InsufficientCapacityError causing a reevaluation, with then a retry recalculation of instances maybe falling back to regular on-demand -1. We call CreateFleet API in certain raise conditions, resulting in an InsufficientCapacityError causing a reevaluation, +2. We call CreateFleet API in certain raise conditions, resulting in an InsufficientCapacityError causing a reevaluation, with then a fallback to on-demand could be selected if Capacity Reservations not available ## FAQ