From 0b1536f2147829b994ed1734f0fe922dce3c83fe Mon Sep 17 00:00:00 2001 From: Jason Deal Date: Wed, 14 Aug 2024 10:45:12 -0700 Subject: [PATCH] docs: v1 NodeClass doc updates (#6720) Co-authored-by: Nick Tran <10810510+njtran@users.noreply.github.com> --- .../en/preview/concepts/nodeclasses.md | 113 ++++++++++++------ .../en/preview/upgrading/v1-migration.md | 13 +- 2 files changed, 82 insertions(+), 44 deletions(-) diff --git a/website/content/en/preview/concepts/nodeclasses.md b/website/content/en/preview/concepts/nodeclasses.md index 13df18921c90..854c02a7f63c 100644 --- a/website/content/en/preview/concepts/nodeclasses.md +++ b/website/content/en/preview/concepts/nodeclasses.md @@ -97,15 +97,17 @@ spec: # Each term in the array of amiSelectorTerms is ORed together # Within a single term, all conditions are ANDed amiSelectorTerms: - # Select on any AMI that has both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag - # AND the "environment: test" tag OR any AMI with the "my-ami" name - # OR any AMI with ID "ami-123" - - alias: al2023@v20240625 # Use alias to select a particular EKS optimized AMI + # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}` + # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with + # ID `ami-123` - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 + # Select EKS optimized AL2023 AMIs with version `v20240703`. This term is mutually + # exclusive and can't be specified with other terms. + # - alias: al2023@v20240703 # Optional, propagates tags to underlying EC2 resources tags: @@ -209,9 +211,11 @@ Refer to the [NodePool docs]({{}}) for settings applicable t ## spec.kubelet -Karpenter provides the ability to specify a few additional Kubelet args. These are all optional and provide support for -additional customization and use cases. Adjust these only if you know you need to do so. For more details on kubelet configuration arguments, [see the KubeletConfiguration API specification docs](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1/). -The implemented fields are a subset of the full list of upstream kubelet configuration arguments. Please cut an issue if you'd like to see another field implemented. +Karpenter provides the ability to specify a few additional Kubelet arguments. +These are all optional and provide support for additional customization and use cases. +Adjust these only if you know you need to do so. +For more details on kubelet settings, see the [KubeletConfiguration reference](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1/). +The implemented fields are a subset of the full list of upstream kubelet configuration arguments. ```yaml kubelet: @@ -244,6 +248,44 @@ kubelet: clusterDNS: ["10.0.1.100"] ``` +{{% alert title="Note" color="primary" %}} +If you need to specify a field that isn't present in `spec.kubelet`, you can set it via custom [UserData]({{< ref "#specuserdata" >}}). +For example, if you wanted to configure `maxPods` and `registryPullQPS` you would set the former through `spec.kubelet` and the latter through UserData. +The following example achieves this with AL2023: + +```yaml +apiVersion: karpenter.k8s.aws/v1 +kind: EC2NodeClass +spec: + amiSelectorTerms: + - alias: al2023@v20240807 + kubelet: + maxPods: 42 + userData: | + apiVersion: node.eks.aws/v1alpha1 + kind: NodeConfig + spec: + kubelet: + config: + # Configured through UserData since unavailable in `spec.kubelet` + registryPullQPS: 10 +``` + +Note that when using the `Custom` AMIFamily you will need to specify fields **both** in `spec.kublet` and `spec.userData`. +{{% /alert %}} + +{{% alert title="Warning" color="warning" %}} +The Bottlerocket AMIFamily does not support the following fields: + +* `evictionSoft` +* `evictionSoftGracePeriod` +* `evictionMaxPodGracePeriod` +* `cpuCFSQuota` + +If any of these fields are specified on a Bottlerocket EC2NodeClass, they will be ommited from generated UserData and ignored for scheduling purposes. +Support for these fields can be tracked via GitHub issue [#3722](https://github.com/aws/karpenter-provider-aws/issues/3722). +{{% /alert %}} + #### Pods Per Core An alternative way to dynamically set the maximum density of pods on a node is to use the `.spec.kubelet.podsPerCore` value. Karpenter will calculate the pod density during scheduling by multiplying this value by the number of logical cores (vCPUs) on an instance type. This value will also be passed through to the `--pods-per-core` value on kubelet startup to configure the number of allocatable pods the kubelet can assign to the node instance. @@ -254,10 +296,6 @@ The value generated from `podsPerCore` cannot exceed `maxPods`, meaning, if both `maxPods` may not be set in the `kubelet` of an EC2NodeClass, but may still be restricted by the `ENI_LIMITED_POD_DENSITY` value. You may want to ensure that the `podsPerCore` value that will be used for instance families associated with the EC2NodeClass will not cause unexpected behavior by exceeding the `maxPods` value. {{% /alert %}} -{{% alert title="Pods Per Core on Bottlerocket" color="warning" %}} -Bottlerocket AMIFamily currently does not support `podsPerCore` configuration. If a EC2NodeClass contains a `provider` or `providerRef` to a node template that will launch a Bottlerocket instance, the `podsPerCore` value will be ignored for scheduling and for configuring the kubelet. -{{% /alert %}} - #### Max Pods For small instances that require an increased pod density or large instances that require a reduced pod density, you can override this default value with `.spec.kubelet.maxPods`. This value will be used during Karpenter pod scheduling and passed through to `--max-pods` on kubelet startup. @@ -356,11 +394,10 @@ It's currently not possible to specify custom networking with Windows nodes. AMIFamily dictates the default bootstrapping logic for nodes provisioned through this `EC2NodeClass`. An `amiFamily` is only required if you don't specify a `spec.amiSelectorTerms.alias` object. -For example, if you specify `alias: al2023@v20240625`, the `amiFamily` is implicitly `AL2023`. +For example, if you specify `alias: al2023@v20240807`, the `amiFamily` is implicitly `AL2023`. AMIFamily does not impact which AMI is discovered, only the UserData generation and default BlockDeviceMappings. To automatically discover EKS optimized AMIs, use the new [`alias` field in amiSelectorTerms]({{< ref "#specamiselectorterms" >}}). - {{% alert title="Ubuntu Support Dropped at v1" color="warning" %}} Support for the Ubuntu AMIFamily has been dropped at Karpenter `v1.0.0`. @@ -471,10 +508,6 @@ max-pods = 110 ``` -{{% alert title="Note" color="primary" %}} -Karpenter will automatically query for the appropriate [EKS optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-amis.html) via AWS Systems Manager (SSM). In the case of the `Custom` AMIFamily, no default AMIs are defined. As a result, `amiSelectorTerms` must be specified to inform Karpenter on which custom AMIs are to be used. -{{% /alert %}} - ### Custom The `Custom` AMIFamily ships without any default userData to allow you to configure custom bootstrapping for control planes or images that don't support the default methods from the other families. For this AMIFamily, kubelet must add the taint `karpenter.sh/unregistered:NoExecute` via the `--register-with-taints` flag ([flags](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#options)) or the KubeletConfiguration spec ([options](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1/#kubelet-config-k8s-io-v1-CredentialProviderConfig) and [docs](https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/)). Karpenter will fail to register nodes that do not have this taint. @@ -659,24 +692,28 @@ For [private clusters](https://docs.aws.amazon.com/eks/latest/userguide/private- AMI Selector Terms are __required__ and are used to configure AMIs for Karpenter to use. AMIs are discovered through alias, id, owner, name, and [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). -This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. Effectively, all requirements within a single term are ANDed together. It's possible that you may want to select on two different AMIs that have unrelated requirements. In this case, you can specify multiple terms which will be ORed together to form your selection logic. The example below shows how this selection logic is fulfilled. +This selection logic is modeled as terms, where each term contains multiple conditions that must all be satisfied for the selector to match. +Effectively, all requirements within a single term are ANDed together. +It's possible that you may want to select on two different AMIs that have unrelated requirements. +In this case, you can specify multiple terms which will be ORed together to form your selection logic. +The example below shows how this selection logic is fulfilled. ```yaml amiSelectorTerms: - # Select on any AMI that has an al2023 AMI family and 20240625 version, - # and both the "karpenter.sh/discovery: ${CLUSTER_NAME}" tag - # AND the "environment: test" tag OR any AMI with the "my-ami" name - # OR any AMI with ID "ami-123" - - alias: al2023@v20240625 + # Select on any AMI that has both the `karpenter.sh/discovery: ${CLUSTER_NAME}` + # AND `environment: test` tags OR any AMI with the name `my-ami` OR an AMI with + # ID `ami-123` - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" environment: test - name: my-ami - id: ami-123 + # Select EKS optimized AL2023 AMIs with version `v20240807`. This term is mutually + # exclusive and can't be specified with other terms. + # - alias: al2023@v20240807 ``` -An `alias` has the following format: `family@version`. -Use the `alias` field to select an EKS-optimized AMI family and version. Family can be one of the following values: +An `alias` term can be used to select EKS-optimized AMIs. An `alias` is formatted as `family@version`. Family can be one of the following values: * `al2` * `al2023` @@ -686,17 +723,15 @@ Use the `alias` field to select an EKS-optimized AMI family and version. Family The version string can be set to `latest`, or pinned to a specific AMI using the format of that AMI's GitHub release tags. For example, AL2 and AL2023 use dates for their release, so they can be pinned as follows: -``` +```yaml alias: al2023@v20240703 ``` Bottlerocket uses a semantic version for their releases. You can pin bottlerocket as follows: -``` +```yaml alias: bottlerocket@v1.20.4 ``` The Windows family does not support pinning, so only `latest` is supported. -An `alias` is mutually exclusive and may not be specified with any other terms. - To select an AMI by name, use the `name` field in the selector term. To select an AMI by id, use the `id` field in the selector term. To select AMIs that are not owned by `amazon` or the account that Karpenter is running in, use the `owner` field - you can use a combination of account aliases (e.g. `self` `amazon`, `your-aws-account-name`) and account IDs. If owner is not set for `name`, it defaults to `self,amazon`, preventing Karpenter from inadvertently selecting an AMI that is owned by a different account. Tags don't require an owner as tags can only be discovered by the user who created them. @@ -709,7 +744,7 @@ AMIs may be specified by any AWS tag, including `Name`. Selecting by tag or by n If `amiSelectorTerms` match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints: * When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType's requirements. - * Note that Karpenter **cannot** detect any requirement other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate `EC2NodeClass`. + * Unless using an alias, Karpenter **cannot** detect requirements other than architecture. If you need to specify different AMIs for different kind of nodes (e.g. accelerated GPU AMIs), you should use a separate `EC2NodeClass`. * If multiple AMIs are found that can be used, Karpenter will choose the latest one. * If no AMIs are found that can be used, then no nodes will be provisioned. {{% /alert %}} @@ -719,7 +754,7 @@ If `amiSelectorTerms` match more than one AMI, Karpenter will automatically dete Select by AMI family and version: ```yaml amiSelectorTerms: - - alias: al2023@v20240625 + - alias: al2023@v20240807 ``` Select all with a specified tag: @@ -1053,8 +1088,8 @@ exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 --//-- ``` -{{% alert title="Note" color="primary" %}} -You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet: +{{% alert title="Tip" color="secondary" %}} +You can set additional kubelet configuration properties, unavailable through `spec.kubelet`, by updating the `kubelet-config.json` file: ```yaml apiVersion: karpenter.k8s.aws/v1 @@ -1062,7 +1097,6 @@ kind: EC2NodeClass metadata: name: kubelet-config-example spec: - ... amiFamily: AL2 userData: | #!/bin/bash @@ -1325,6 +1359,9 @@ spec: ### Custom * No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster. +* Custom UserData must meet the following requirements to work correctly with Karpenter: + * It must ensure the node is registered with the `karpenter.sh/unregistered:NoExecute` taint (via kubelet configuration field `registerWithTaints`) + * It must set kubelet config options to match those configured in `spec.kubelet` ## spec.detailedMonitoring @@ -1398,11 +1435,12 @@ status: #### Examples -Default AMIs resolved from the AL2 AMIFamily: +AMIs resolved with an AL2 alias: ```yaml spec: - amiFamily: AL2 + amiSelectorTerms: + - alias: al2@v20240807 status: amis: - id: ami-03c3a3dcda64f5b75 @@ -1447,11 +1485,10 @@ status: operator: DoesNotExist ``` -AMIs resolved from [`spec.amiSelectorTerms`]({{< ref "#specamiselectorterms" >}}): +AMIs resolved from tags: ```yaml spec: - amiFamily: AL2 amiSelectorTerms: - tags: karpenter.sh/discovery: "${CLUSTER_NAME}" diff --git a/website/content/en/preview/upgrading/v1-migration.md b/website/content/en/preview/upgrading/v1-migration.md index b18336cabacf..0b5d6789cd7b 100644 --- a/website/content/en/preview/upgrading/v1-migration.md +++ b/website/content/en/preview/upgrading/v1-migration.md @@ -25,7 +25,7 @@ See the [Changelog]({{}}) for details about actions you shoul Please read through the entire procedure before beginning the upgrade. There are major changes in this upgrade, so please evaluate the list of breaking changes before continuing. {{% alert title="Note" color="warning" %}} -The upgrade guide will first require upgrading to your latest patch version prior to upgrade to v1.0.0. This will be to allow the conversion webhooks to operate and minimize downtime of the Karpenter controller when requesting the Karpenter custom resources. +The upgrade guide will first require upgrading to your latest patch version prior to upgrade to v1.0.0. This will be to allow the conversion webhooks to operate and minimize downtime of the Karpenter controller when requesting the Karpenter custom resources. {{% /alert %}} 1. Set environment variables for your cluster to upgrade to the latest patch version of the current Karpenter version you're running on: @@ -80,7 +80,7 @@ The upgrade guide will first require upgrading to your latest patch version prio 7. Upgrade Karpenter to the latest patch version of your current minor version's. At the end of this step, conversion webhooks will run but will not convert any version. ```bash - # Service account annotation can be dropped when using pod identity + # Service account annotation can be dropped when using pod identity helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ --set settings.clusterName=${CLUSTER_NAME} \ @@ -127,7 +127,7 @@ The upgrade guide will first require upgrading to your latest patch version prio 11. Upgrade Karpenter to the new version. At the end of this step, conversion webhooks run to convert the Karpenter CRDs to v1. ```bash - # Service account annotion can be dropped when using pod identity + # Service account annotion can be dropped when using pod identity helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ --set settings.clusterName=${CLUSTER_NAME} \ @@ -251,7 +251,7 @@ Keep in mind that rollback, without replacing the Karpenter nodes, will not be s ### Downgrading -Once the Karpenter CRDs are upgraded to v1, conversion webhooks are needed to help convert APIs that are stored in etcd from v1 to v1beta1. Also changes to the CRDs will need to at least include the latest version of the CRD in this case being v1. The patch versions of the v1beta1 Karpenter controller that include the conversion wehooks include: +Once the Karpenter CRDs are upgraded to v1, conversion webhooks are needed to help convert APIs that are stored in etcd from v1 to v1beta1. Also changes to the CRDs will need to at least include the latest version of the CRD in this case being v1. The patch versions of the v1beta1 Karpenter controller that include the conversion wehooks include: * v0.37.1 * v0.36.3 @@ -308,10 +308,10 @@ helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-cr --set webhook.port=8443 ``` -4. Rollback the Karpenter Controller +4. Rollback the Karpenter Controller ```bash -# Service account annotation can be dropped when using pod identity +# Service account annotation can be dropped when using pod identity helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ --set settings.clusterName=${CLUSTER_NAME} \ @@ -338,6 +338,7 @@ Karpenter should now be pulling and operating against the v1beta1 APIVersion as * Expiration is now forceful and begins draining as soon as it’s expired. Karpenter does not wait for replacement capacity to be available before draining, but will start provisioning a replacement as soon as the node is expired and begins draining. * Karpenter's generated NodeConfig now takes precedence when generating UserData with the AL2023 `amiFamily`. If you're setting any values managed by Karpenter in your AL2023 UserData, configure these through Karpenter natively (e.g. kubelet configuration fields). * Karpenter now adds a `karpenter.sh/unregistered:NoExecute` taint to nodes in injected UserData when using alias in AMISelectorTerms or non-Custom AMIFamily. When using `amiFamily: Custom`, users will need to add this taint into their UserData, where Karpenter will automatically remove it when provisioning nodes. + * Discovered standard AL2023 AMIs will no longer be considered compatible with GPU / accelerator workloads. If you're using an AL2023 EC2NodeClass (without AMISelectorTerms) for these workloads, you will need to select your AMI via AMISelectorTerms (non-alias). * API Moves: * ExpireAfter has moved from the `NodePool.Spec.Disruption` block to `NodePool.Spec.Template.Spec`, and is now a drift-able field. * `Kubelet` was moved to the EC2NodeClass from the NodePool.