diff --git a/charts/karpenter/README.md b/charts/karpenter/README.md index 694a517d720f..791fc20e8dd5 100644 --- a/charts/karpenter/README.md +++ b/charts/karpenter/README.md @@ -72,9 +72,6 @@ cosign verify public.ecr.aws/karpenter/karpenter:1.0.0 \ | podDisruptionBudget.name | string | `"karpenter"` | | | podLabels | object | `{}` | Additional labels for the pod. | | podSecurityContext | object | `{"fsGroup":65532}` | SecurityContext for the pod. | -| postInstallHook.image.digest | string | `"sha256:13a2ad1bd37ce42ee2a6f1ab0d30595f42eb7fe4a90d6ec848550524104a1ed6"` | SHA256 digest of the post-install hook image. | -| postInstallHook.image.repository | string | `"public.ecr.aws/bitnami/kubectl"` | Repository path to the post-install hook. This minimally needs to have `kubectl` installed | -| postInstallHook.image.tag | string | `"1.30"` | Tag of the post-install hook image. | | priorityClassName | string | `"system-cluster-critical"` | PriorityClass name for the pod. | | replicas | int | `2` | Number of replicas. | | revisionHistoryLimit | int | `10` | The number of old ReplicaSets to retain to allow rollback. | @@ -100,9 +97,6 @@ cosign verify public.ecr.aws/karpenter/karpenter:1.0.0 \ | terminationGracePeriodSeconds | string | `nil` | Override the default termination grace period for the pod. | | tolerations | list | `[{"key":"CriticalAddonsOnly","operator":"Exists"}]` | Tolerations to allow the pod to be scheduled to nodes with taints. | | topologySpreadConstraints | list | `[{"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"DoNotSchedule"}]` | Topology spread constraints to increase the controller resilience by distributing pods across the cluster zones. If an explicit label selector is not provided one will be created from the pod selector labels. | -| webhook.enabled | bool | `true` | Whether to enable the webhooks and webhook permissions. | -| webhook.metrics.port | int | `8001` | The container port to use for webhook metrics. | -| webhook.port | int | `8443` | The container port to use for the webhook. | ---------------------------------------------- diff --git a/website/content/en/docs/upgrading/upgrade-guide.md b/website/content/en/docs/upgrading/upgrade-guide.md index 3259045631dd..b74ac48c4bac 100644 --- a/website/content/en/docs/upgrading/upgrade-guide.md +++ b/website/content/en/docs/upgrading/upgrade-guide.md @@ -89,6 +89,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced * Karpenter no longer updates the logger name when creating controller loggers. We now adhere to the controller-runtime standard, where the logger name will be set as `"logger": "controller"` always and the controller name will be stored in the structured value `"controller"` * Karpenter updated the NodeClass controller naming in the following way: `nodeclass` -> `nodeclass.status`, `nodeclass.hash`, `nodeclass.termination` * Karpenter's NodeClaim status conditions no longer include the `severity` field +* Starting with `0.37.3` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.36.0`+ @@ -101,6 +102,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced {{% /alert %}} * Karpenter changed the name of the `karpenter_cloudprovider_instance_type_price_estimate` metric to `karpenter_cloudprovider_instance_type_offering_price_estimate` to align with the new `karpenter_cloudprovider_instance_type_offering_available` metric. The `region` label was also dropped from the metric, since this can be inferred from the environment that Karpenter is running in. +* Starting with `0.36.5` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.35.0`+ @@ -109,6 +111,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced {{% /alert %}} * Karpenter OCI tags and Helm chart version are now valid semantic versions, meaning that the `v` prefix from the git tag has been removed and they now follow the `x.y.z` pattern. +* Starting with `0.35.8` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.34.0`+ @@ -129,6 +132,7 @@ The Ubuntu EKS optimized AMI has moved from 20.04 to 22.04 for Kubernetes 1.29+. * Karpenter now adds a default `podSecurityContext` that configures the `fsgroup: 65536` of volumes in the pod. If you are using sidecar containers, you should review if this configuration is compatible for them. You can disable this default `podSecurityContext` through helm by performing `--set podSecurityContext=null` when installing/upgrading the chart. * The `dnsPolicy` for the Karpenter controller pod has been changed back to the Kubernetes cluster default of `ClusterFirst`. Setting our `dnsPolicy` to `Default` (confusingly, this is not the Kubernetes cluster default) caused more confusion for any users running IPv6 clusters with dual-stack nodes or anyone running Karpenter with dependencies on cluster services (like clusters running service meshes). This change may be breaking for any users on Fargate or MNG who were allowing Karpenter to manage their in-cluster DNS service (`core-dns` on most clusters). If you still want the old behavior here, you can change the `dnsPolicy` to point to use `Default` by setting the helm value on install/upgrade with `--set dnsPolicy=Default`. More details on this issue can be found in the following Github issues: [#2186](https://github.com/aws/karpenter-provider-aws/issues/2186) and [#4947](https://github.com/aws/karpenter-provider-aws/issues/4947). * Karpenter now disallows `nodepool.spec.template.spec.resources` to be set. The webhook validation never allowed `nodepool.spec.template.spec.resources`. We are now ensuring that CEL validation also disallows `nodepool.spec.template.spec.resources` to be set. If you were previously setting the resources field on your NodePool, ensure that you remove this field before upgrading to the newest version of Karpenter or else updates to the resource may fail on the new version. +* Starting with `0.34.9` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.33.0`+ @@ -145,6 +149,7 @@ The Ubuntu EKS optimized AMI has moved from 20.04 to 22.04 for Kubernetes 1.29+. * `0.33.0` drops looking up the `zap-logger-config` through ConfigMap discovery. Instead, Karpenter now expects the logging config to be mounted on the filesystem if you are using this to configure Zap logging. This is not enabled by default, but can be enabled through `--set logConfig.enabled=true` in the Helm values. If you are setting any values in the `logConfig` from the `0.32.x` upgrade, such as `logConfig.logEncoding`, note that you will have to explicitly set `logConfig.enabled=true` alongside it. Also, note that setting the Zap logging config is a deprecated feature in beta and is planned to be dropped at v1. View the [Logging Configuration Section of the v1beta1 Migration Guide]({{}}) for more details. * `0.33.0` change the default `LOG_LEVEL` from `debug` to `info` by default. If you are still enabling logging configuration through the `zap-logger-config`, no action is required. * `0.33.0` drops support for comma delimited lists on tags for `SubnetSelectorTerm`, `SecurityGroupsSelectorTerm`, and `AMISelectorTerm`. Karpenter now supports multiple terms for each of the selectors which means that we can specify a more explicit OR-based constraint through separate terms rather than a comma-delimited list of values. +* Starting with `0.33.8` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.32.0`+ diff --git a/website/content/en/docs/upgrading/v1-migration.md b/website/content/en/docs/upgrading/v1-migration.md index fd1eb7944308..15a086986920 100644 --- a/website/content/en/docs/upgrading/v1-migration.md +++ b/website/content/en/docs/upgrading/v1-migration.md @@ -52,11 +52,11 @@ The upgrade guide will first require upgrading to your latest patch version prio The Karpenter version you are running must be between minor version `v0.33` and `v0.37`. To be able to roll back from Karpenter v1, you must rollback to on the following patch release versions for your minor version, which will include the conversion webhooks for a smooth rollback: - * v0.37.2 - * v0.36.4 - * v0.35.7 - * v0.34.8 - * v0.33.7 + * v0.37.3 + * v0.36.5 + * v0.35.8 + * v0.34.9 + * v0.33.8 3. Review for breaking changes between v0.33 and v0.37: If you are already running Karpenter v0.37.x, you can skip this step. If you are running an earlier Karpenter version, you need to review the [Upgrade Guide]({{}}) for each minor release. @@ -291,11 +291,11 @@ Keep in mind that rollback, without replacing the Karpenter nodes, will not be s Once the Karpenter CRDs are upgraded to v1, conversion webhooks are needed to help convert APIs that are stored in etcd from v1 to v1beta1. Also changes to the CRDs will need to at least include the latest version of the CRD in this case being v1. The patch versions of the v1beta1 Karpenter controller that include the conversion wehooks include: -* v0.37.2 -* v0.36.4 -* v0.35.7 -* v0.34.8 -* v0.33.7 +* v0.37.3 +* v0.36.5 +* v0.35.8 +* v0.34.9 +* v0.33.8 {{% alert title="Note" color="warning" %}} When rolling back from v1, Karpenter will not retain data that was only valid in v1 APIs. For instance, if you were upgrading from v0.33.5 to v1, updated the `NodePool.Spec.Disruption.Budgets` field and then rolled back to v0.33.6, Karpenter would not retain the `NodePool.Spec.Disruption.Budgets` field, as that was introduced in v0.34.x. If you are configuring the kubelet field, and have removed the `compatibility.karpenter.sh/v1beta1-kubelet-conversion` annotation, rollback is not supported without replacing your nodes between EC2NodeClass and NodePool. @@ -319,7 +319,7 @@ export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/ 2. Set Karpenter Version ```bash -# Note: v0.33.6 and v0.34.7 include the v prefix, omit it for versions v0.35+ +# Note: v0.33.x and v0.34.x include the v prefix, omit it for versions v0.35+ export KARPENTER_VERSION="" ``` diff --git a/website/content/en/preview/reference/instance-types.md b/website/content/en/preview/reference/instance-types.md index d11285618aee..0f1c0c036e2a 100644 --- a/website/content/en/preview/reference/instance-types.md +++ b/website/content/en/preview/reference/instance-types.md @@ -3874,6 +3874,151 @@ below are the resources available with some assumptions and after the instance o |pods|737| |vpc.amazonaws.com/efa|1| |vpc.amazonaws.com/pod-eni|107| +## c7g-flex Family +### `c7g-flex.medium` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|1| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|2048| + |karpenter.k8s.aws/instance-size|medium| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.medium| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|940m| + |ephemeral-storage|17Gi| + |memory|1392Mi| + |pods|8| +### `c7g-flex.large` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|2| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|4096| + |karpenter.k8s.aws/instance-size|large| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.large| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|1930m| + |ephemeral-storage|17Gi| + |memory|3055Mi| + |pods|29| +### `c7g-flex.xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|4| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|8192| + |karpenter.k8s.aws/instance-size|xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|3920m| + |ephemeral-storage|17Gi| + |memory|6525Mi| + |pods|58| +### `c7g-flex.2xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|8| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|16384| + |karpenter.k8s.aws/instance-size|2xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.2xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|7910m| + |ephemeral-storage|17Gi| + |memory|14103Mi| + |pods|58| +### `c7g-flex.4xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|16| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|32768| + |karpenter.k8s.aws/instance-size|4xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.4xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|15890m| + |ephemeral-storage|17Gi| + |memory|27322Mi| + |pods|234| +### `c7g-flex.8xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|c| + |karpenter.k8s.aws/instance-cpu|32| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|c7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|65536| + |karpenter.k8s.aws/instance-size|8xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|c7g-flex.8xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|31850m| + |ephemeral-storage|17Gi| + |memory|57632Mi| + |pods|234| ## c7gd Family ### `c7gd.medium` #### Labels @@ -4672,6 +4817,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|3114Mi| |pods|29| + |vpc.amazonaws.com/pod-eni|3| ### `c7i-flex.xlarge` #### Labels | Label | Value | @@ -4697,6 +4843,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|6584Mi| |pods|58| + |vpc.amazonaws.com/pod-eni|8| ### `c7i-flex.2xlarge` #### Labels | Label | Value | @@ -4722,6 +4869,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|14162Mi| |pods|58| + |vpc.amazonaws.com/pod-eni|18| ### `c7i-flex.4xlarge` #### Labels | Label | Value | @@ -4747,6 +4895,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|27381Mi| |pods|234| + |vpc.amazonaws.com/pod-eni|34| ### `c7i-flex.8xlarge` #### Labels | Label | Value | @@ -4772,6 +4921,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|57691Mi| |pods|234| + |vpc.amazonaws.com/pod-eni|54| ## d2 Family ### `d2.xlarge` #### Labels @@ -5315,7 +5465,6 @@ below are the resources available with some assumptions and after the instance o |memory|112629Mi| |nvidia.com/gpu|1| |pods|234| - |vpc.amazonaws.com/pod-eni|6| ### `g3.8xlarge` #### Labels | Label | Value | @@ -5346,7 +5495,6 @@ below are the resources available with some assumptions and after the instance o |memory|228187Mi| |nvidia.com/gpu|2| |pods|234| - |vpc.amazonaws.com/pod-eni|6| ### `g3.16xlarge` #### Labels | Label | Value | @@ -5407,7 +5555,6 @@ below are the resources available with some assumptions and after the instance o |memory|27896Mi| |nvidia.com/gpu|1| |pods|58| - |vpc.amazonaws.com/pod-eni|10| ## g4ad Family ### `g4ad.xlarge` #### Labels @@ -6541,6 +6688,7 @@ below are the resources available with some assumptions and after the instance o |memory|29317Mi| |nvidia.com/gpu|1| |pods|58| + |vpc.amazonaws.com/pod-eni|18| ### `g6e.2xlarge` #### Labels | Label | Value | @@ -6572,6 +6720,7 @@ below are the resources available with some assumptions and after the instance o |memory|59627Mi| |nvidia.com/gpu|1| |pods|58| + |vpc.amazonaws.com/pod-eni|38| ### `g6e.4xlarge` #### Labels | Label | Value | @@ -6603,6 +6752,7 @@ below are the resources available with some assumptions and after the instance o |memory|118312Mi| |nvidia.com/gpu|1| |pods|234| + |vpc.amazonaws.com/pod-eni|54| ### `g6e.8xlarge` #### Labels | Label | Value | @@ -6635,6 +6785,7 @@ below are the resources available with some assumptions and after the instance o |nvidia.com/gpu|1| |pods|234| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|84| ### `g6e.12xlarge` #### Labels | Label | Value | @@ -6667,6 +6818,7 @@ below are the resources available with some assumptions and after the instance o |nvidia.com/gpu|4| |pods|292| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|112| ### `g6e.16xlarge` #### Labels | Label | Value | @@ -6699,6 +6851,7 @@ below are the resources available with some assumptions and after the instance o |nvidia.com/gpu|1| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ### `g6e.24xlarge` #### Labels | Label | Value | @@ -6731,6 +6884,7 @@ below are the resources available with some assumptions and after the instance o |nvidia.com/gpu|4| |pods|492| |vpc.amazonaws.com/efa|2| + |vpc.amazonaws.com/pod-eni|102| ### `g6e.48xlarge` #### Labels | Label | Value | @@ -6763,6 +6917,7 @@ below are the resources available with some assumptions and after the instance o |nvidia.com/gpu|8| |pods|492| |vpc.amazonaws.com/efa|4| + |vpc.amazonaws.com/pod-eni|82| ## gr6 Family ### `gr6.4xlarge` #### Labels @@ -12881,6 +13036,151 @@ below are the resources available with some assumptions and after the instance o |pods|737| |vpc.amazonaws.com/efa|1| |vpc.amazonaws.com/pod-eni|107| +## m7g-flex Family +### `m7g-flex.medium` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|1| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|4096| + |karpenter.k8s.aws/instance-size|medium| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.medium| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|940m| + |ephemeral-storage|17Gi| + |memory|3286Mi| + |pods|8| +### `m7g-flex.large` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|2| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|8192| + |karpenter.k8s.aws/instance-size|large| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.large| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|1930m| + |ephemeral-storage|17Gi| + |memory|6844Mi| + |pods|29| +### `m7g-flex.xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|4| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|16384| + |karpenter.k8s.aws/instance-size|xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|3920m| + |ephemeral-storage|17Gi| + |memory|14103Mi| + |pods|58| +### `m7g-flex.2xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|8| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|32768| + |karpenter.k8s.aws/instance-size|2xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.2xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|7910m| + |ephemeral-storage|17Gi| + |memory|29258Mi| + |pods|58| +### `m7g-flex.4xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|16| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|65536| + |karpenter.k8s.aws/instance-size|4xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.4xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|15890m| + |ephemeral-storage|17Gi| + |memory|57632Mi| + |pods|234| +### `m7g-flex.8xlarge` +#### Labels + | Label | Value | + |--|--| + |karpenter.k8s.aws/instance-category|m| + |karpenter.k8s.aws/instance-cpu|32| + |karpenter.k8s.aws/instance-cpu-manufacturer|aws| + |karpenter.k8s.aws/instance-ebs-bandwidth|10000| + |karpenter.k8s.aws/instance-encryption-in-transit-supported|true| + |karpenter.k8s.aws/instance-family|m7g-flex| + |karpenter.k8s.aws/instance-generation|7| + |karpenter.k8s.aws/instance-hypervisor|nitro| + |karpenter.k8s.aws/instance-memory|131072| + |karpenter.k8s.aws/instance-size|8xlarge| + |kubernetes.io/arch|arm64| + |kubernetes.io/os|linux| + |node.kubernetes.io/instance-type|m7g-flex.8xlarge| +#### Resources + | Resource | Quantity | + |--|--| + |cpu|31850m| + |ephemeral-storage|17Gi| + |memory|118253Mi| + |pods|234| ## m7gd Family ### `m7gd.medium` #### Labels @@ -18936,6 +19236,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|7075Mi| |pods|8| + |vpc.amazonaws.com/pod-eni|4| ### `r8g.large` #### Labels | Label | Value | @@ -18961,6 +19262,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|14422Mi| |pods|29| + |vpc.amazonaws.com/pod-eni|9| ### `r8g.xlarge` #### Labels | Label | Value | @@ -18986,6 +19288,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|29258Mi| |pods|58| + |vpc.amazonaws.com/pod-eni|18| ### `r8g.2xlarge` #### Labels | Label | Value | @@ -19011,6 +19314,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|59568Mi| |pods|58| + |vpc.amazonaws.com/pod-eni|38| ### `r8g.4xlarge` #### Labels | Label | Value | @@ -19036,6 +19340,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|118253Mi| |pods|234| + |vpc.amazonaws.com/pod-eni|54| ### `r8g.8xlarge` #### Labels | Label | Value | @@ -19061,6 +19366,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|239495Mi| |pods|234| + |vpc.amazonaws.com/pod-eni|54| ### `r8g.12xlarge` #### Labels | Label | Value | @@ -19086,6 +19392,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|360736Mi| |pods|234| + |vpc.amazonaws.com/pod-eni|54| ### `r8g.16xlarge` #### Labels | Label | Value | @@ -19111,6 +19418,7 @@ below are the resources available with some assumptions and after the instance o |ephemeral-storage|17Gi| |memory|476445Mi| |pods|737| + |vpc.amazonaws.com/pod-eni|107| ### `r8g.24xlarge` #### Labels | Label | Value | @@ -19137,6 +19445,7 @@ below are the resources available with some assumptions and after the instance o |memory|718928Mi| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ### `r8g.metal-24xl` #### Labels | Label | Value | @@ -19163,6 +19472,7 @@ below are the resources available with some assumptions and after the instance o |memory|718928Mi| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ### `r8g.48xlarge` #### Labels | Label | Value | @@ -19189,6 +19499,7 @@ below are the resources available with some assumptions and after the instance o |memory|1446378Mi| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ### `r8g.metal-48xl` #### Labels | Label | Value | @@ -19215,6 +19526,7 @@ below are the resources available with some assumptions and after the instance o |memory|1446378Mi| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ## t1 Family ### `t1.micro` #### Labels @@ -20235,6 +20547,7 @@ below are the resources available with some assumptions and after the instance o |memory|11630731Mi| |pods|737| |vpc.amazonaws.com/efa|1| + |vpc.amazonaws.com/pod-eni|107| ## u7in-16tb Family ### `u7in-16tb.224xlarge` #### Labels @@ -20262,6 +20575,7 @@ below are the resources available with some assumptions and after the instance o |memory|15514235Mi| |pods|394| |vpc.amazonaws.com/efa|2| + |vpc.amazonaws.com/pod-eni|106| ## u7in-24tb Family ### `u7in-24tb.224xlarge` #### Labels @@ -20289,6 +20603,7 @@ below are the resources available with some assumptions and after the instance o |memory|23273698Mi| |pods|394| |vpc.amazonaws.com/efa|2| + |vpc.amazonaws.com/pod-eni|106| ## u7in-32tb Family ### `u7in-32tb.224xlarge` #### Labels @@ -20316,6 +20631,7 @@ below are the resources available with some assumptions and after the instance o |memory|31033160Mi| |pods|394| |vpc.amazonaws.com/efa|2| + |vpc.amazonaws.com/pod-eni|106| ## vt1 Family ### `vt1.3xlarge` #### Labels diff --git a/website/content/en/preview/reference/metrics.md b/website/content/en/preview/reference/metrics.md index 674dc859179c..58056b914394 100644 --- a/website/content/en/preview/reference/metrics.md +++ b/website/content/en/preview/reference/metrics.md @@ -64,10 +64,6 @@ Number of nodes terminated in total by Karpenter. Labeled by owning nodepool. Node system daemon overhead are the resources reserved for system overhead, the difference between the node's capacity and allocatable values are reported by the status. - Stability Level: BETA -### `karpenter_nodes_leases_deleted_total` -Number of deleted leaked leases. -- Stability Level: ALPHA - ### `karpenter_nodes_created_total` Number of nodes created in total by Karpenter. Labeled by owning nodepool. - Stability Level: STABLE diff --git a/website/content/en/preview/reference/settings.md b/website/content/en/preview/reference/settings.md index a84521d42b34..0f3d4f36565c 100644 --- a/website/content/en/preview/reference/settings.md +++ b/website/content/en/preview/reference/settings.md @@ -18,7 +18,6 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | CLUSTER_ENDPOINT | \-\-cluster-endpoint | The external kubernetes cluster endpoint for new nodes to connect with. If not specified, will discover the cluster endpoint using DescribeCluster API.| | CLUSTER_NAME | \-\-cluster-name | [REQUIRED] The kubernetes cluster name for resource discovery.| | DISABLE_LEADER_ELECTION | \-\-disable-leader-election | Disable the leader election client before executing the main loop. Disable when running replicated components for high availability is not desired.| -| DISABLE_WEBHOOK | \-\-disable-webhook | Disable the conversion webhooks| | ENABLE_PROFILING | \-\-enable-profiling | Enable the profiling on the metric endpoint| | FEATURE_GATES | \-\-feature-gates | Optional features can be enabled / disabled using feature gates. Current options are: SpotToSpotConsolidation (default = SpotToSpotConsolidation=false)| | HEALTH_PROBE_PORT | \-\-health-probe-port | The port the health probe endpoint binds to for reporting controller health (default = 8081)| @@ -34,8 +33,6 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8080)| | RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075)| -| WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| -| WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| [comment]: <> (end docs generated content from hack/docs/configuration_gen_docs.go) diff --git a/website/content/en/preview/upgrading/upgrade-guide.md b/website/content/en/preview/upgrading/upgrade-guide.md index 018223a56bfc..3e014be97c3c 100644 --- a/website/content/en/preview/upgrading/upgrade-guide.md +++ b/website/content/en/preview/upgrading/upgrade-guide.md @@ -89,6 +89,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced * Karpenter no longer updates the logger name when creating controller loggers. We now adhere to the controller-runtime standard, where the logger name will be set as `"logger": "controller"` always and the controller name will be stored in the structured value `"controller"` * Karpenter updated the NodeClass controller naming in the following way: `nodeclass` -> `nodeclass.status`, `nodeclass.hash`, `nodeclass.termination` * Karpenter's NodeClaim status conditions no longer include the `severity` field +* Starting with `0.37.3` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.36.0`+ @@ -101,6 +102,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced {{% /alert %}} * Karpenter changed the name of the `karpenter_cloudprovider_instance_type_price_estimate` metric to `karpenter_cloudprovider_instance_type_offering_price_estimate` to align with the new `karpenter_cloudprovider_instance_type_offering_available` metric. The `region` label was also dropped from the metric, since this can be inferred from the environment that Karpenter is running in. +* Starting with `0.36.5` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.35.0`+ @@ -109,6 +111,7 @@ Below is the full changelog for v1, copied from the [v1 Migration Upgrade Proced {{% /alert %}} * Karpenter OCI tags and Helm chart version are now valid semantic versions, meaning that the `v` prefix from the git tag has been removed and they now follow the `x.y.z` pattern. +* Starting with `0.35.8` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.34.0`+ @@ -129,6 +132,7 @@ The Ubuntu EKS optimized AMI has moved from 20.04 to 22.04 for Kubernetes 1.29+. * Karpenter now adds a default `podSecurityContext` that configures the `fsgroup: 65536` of volumes in the pod. If you are using sidecar containers, you should review if this configuration is compatible for them. You can disable this default `podSecurityContext` through helm by performing `--set podSecurityContext=null` when installing/upgrading the chart. * The `dnsPolicy` for the Karpenter controller pod has been changed back to the Kubernetes cluster default of `ClusterFirst`. Setting our `dnsPolicy` to `Default` (confusingly, this is not the Kubernetes cluster default) caused more confusion for any users running IPv6 clusters with dual-stack nodes or anyone running Karpenter with dependencies on cluster services (like clusters running service meshes). This change may be breaking for any users on Fargate or MNG who were allowing Karpenter to manage their in-cluster DNS service (`core-dns` on most clusters). If you still want the old behavior here, you can change the `dnsPolicy` to point to use `Default` by setting the helm value on install/upgrade with `--set dnsPolicy=Default`. More details on this issue can be found in the following Github issues: [#2186](https://github.com/aws/karpenter-provider-aws/issues/2186) and [#4947](https://github.com/aws/karpenter-provider-aws/issues/4947). * Karpenter now disallows `nodepool.spec.template.spec.resources` to be set. The webhook validation never allowed `nodepool.spec.template.spec.resources`. We are now ensuring that CEL validation also disallows `nodepool.spec.template.spec.resources` to be set. If you were previously setting the resources field on your NodePool, ensure that you remove this field before upgrading to the newest version of Karpenter or else updates to the resource may fail on the new version. +* Starting with `0.34.9` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.33.0`+ @@ -145,6 +149,7 @@ The Ubuntu EKS optimized AMI has moved from 20.04 to 22.04 for Kubernetes 1.29+. * `0.33.0` drops looking up the `zap-logger-config` through ConfigMap discovery. Instead, Karpenter now expects the logging config to be mounted on the filesystem if you are using this to configure Zap logging. This is not enabled by default, but can be enabled through `--set logConfig.enabled=true` in the Helm values. If you are setting any values in the `logConfig` from the `0.32.x` upgrade, such as `logConfig.logEncoding`, note that you will have to explicitly set `logConfig.enabled=true` alongside it. Also, note that setting the Zap logging config is a deprecated feature in beta and is planned to be dropped at v1. View the [Logging Configuration Section of the v1beta1 Migration Guide]({{}}) for more details. * `0.33.0` change the default `LOG_LEVEL` from `debug` to `info` by default. If you are still enabling logging configuration through the `zap-logger-config`, no action is required. * `0.33.0` drops support for comma delimited lists on tags for `SubnetSelectorTerm`, `SecurityGroupsSelectorTerm`, and `AMISelectorTerm`. Karpenter now supports multiple terms for each of the selectors which means that we can specify a more explicit OR-based constraint through separate terms rather than a comma-delimited list of values. +* Starting with `0.33.8` Karpenter has enabled conversion webhooks by default to improve the v1 migration experience. If working with a cluster with a network policy that blocks Ingress, ports 8000, 8001, 8081, 8443 will need to be allowlisted. ### Upgrading to `0.32.0`+ diff --git a/website/content/en/v0.36/getting-started/getting-started-with-karpenter/_index.md b/website/content/en/v0.36/getting-started/getting-started-with-karpenter/_index.md index 82ae722ce5fb..7f785274deaf 100644 --- a/website/content/en/v0.36/getting-started/getting-started-with-karpenter/_index.md +++ b/website/content/en/v0.36/getting-started/getting-started-with-karpenter/_index.md @@ -45,7 +45,7 @@ After setting up the tools, set the Karpenter and Kubernetes version: ```bash export KARPENTER_NAMESPACE="kube-system" -export KARPENTER_VERSION="0.36.4" +export KARPENTER_VERSION="0.36.5" export K8S_VERSION="1.29" ``` @@ -112,13 +112,13 @@ See [Enabling Windows support](https://docs.aws.amazon.com/eks/latest/userguide/ As the OCI Helm chart is signed by [Cosign](https://github.com/sigstore/cosign) as part of the release process you can verify the chart before installing it by running the following command. ```bash -cosign verify public.ecr.aws/karpenter/karpenter:0.36.4 \ +cosign verify public.ecr.aws/karpenter/karpenter:0.36.5 \ --certificate-oidc-issuer=https://token.actions.githubusercontent.com \ --certificate-identity-regexp='https://github\.com/aws/karpenter-provider-aws/\.github/workflows/release\.yaml@.+' \ --certificate-github-workflow-repository=aws/karpenter-provider-aws \ --certificate-github-workflow-name=Release \ - --certificate-github-workflow-ref=refs/tags/v0.36.4 \ - --annotations version=0.36.4 + --certificate-github-workflow-ref=refs/tags/v0.36.5 \ + --annotations version=0.36.5 ``` {{% alert title="DNS Policy Notice" color="warning" %}} diff --git a/website/content/en/v0.36/upgrading/upgrade-guide.md b/website/content/en/v0.36/upgrading/upgrade-guide.md index 3a3905dec00c..9d34151f2f55 100644 --- a/website/content/en/v0.36/upgrading/upgrade-guide.md +++ b/website/content/en/v0.36/upgrading/upgrade-guide.md @@ -28,9 +28,9 @@ If you get the error `invalid ownership metadata; label validation error:` while In general, you can reapply the CRDs in the `crds` directory of the Karpenter Helm chart: ```shell -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.4/pkg/apis/crds/karpenter.sh_nodepools.yaml -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.4/pkg/apis/crds/karpenter.sh_nodeclaims.yaml -kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.4/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.5/pkg/apis/crds/karpenter.sh_nodepools.yaml +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.5/pkg/apis/crds/karpenter.sh_nodeclaims.yaml +kubectl apply -f https://raw.githubusercontent.com/aws/karpenter/v0.36.5/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml ```