From efec6c9e1fc1e9f79fedadb390817b5355daf110 Mon Sep 17 00:00:00 2001 From: Jonathan Innis Date: Fri, 6 Oct 2023 12:01:54 -0700 Subject: [PATCH] docs: Update `NodeTemplate` doc to `NodeClasses` and rename files to v1beta1 (#4761) --- website/content/en/preview/_index.md | 2 +- website/content/en/preview/concepts/_index.md | 2 +- .../{deprovisioning.md => disruption.md} | 8 +- .../en/preview/concepts/node-templates.md | 693 -------------- .../en/preview/concepts/nodeclasses.md | 883 ++++++++++++++++++ .../{provisioners.md => nodepools.md} | 8 +- .../content/en/preview/concepts/scheduling.md | 2 +- website/content/en/preview/faq.md | 20 +- .../getting-started-with-karpenter/_index.md | 4 +- website/content/en/preview/troubleshooting.md | 10 +- website/content/en/preview/upgrade-guide.md | 8 +- website/static/_redirects | 3 + 12 files changed, 918 insertions(+), 725 deletions(-) rename website/content/en/preview/concepts/{deprovisioning.md => disruption.md} (98%) delete mode 100644 website/content/en/preview/concepts/node-templates.md create mode 100644 website/content/en/preview/concepts/nodeclasses.md rename website/content/en/preview/concepts/{provisioners.md => nodepools.md} (99%) diff --git a/website/content/en/preview/_index.md b/website/content/en/preview/_index.md index 93ddf9368a25..9ded256df2a6 100755 --- a/website/content/en/preview/_index.md +++ b/website/content/en/preview/_index.md @@ -24,7 +24,7 @@ Provisioner-level constraints related to Kubernetes and your cloud provider (AWS - Taints (`taints`): Identify taints to add to provisioned nodes. If a pod doesn't have a matching toleration for the taint, the effect set by the taint occurs (NoSchedule, PreferNoSchedule, or NoExecute). See Kubernetes [Taints and Tolerations](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) for details. - Labels (`labels`): Apply arbitrary key-value pairs to nodes that can be matched by pods. - - Requirements (`requirements`): Set acceptable (`In`) and unacceptable (`Out`) Kubernetes and Karpenter values for node provisioning based on [Well-Known Labels](https://kubernetes.io/docs/reference/labels-annotations-taints/) and [cloud-specific settings]({{}}). These can include [instance types](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), [zones](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone), [computer architecture](https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetes-io-arch), and [capacity type]({{}}) (such as AWS spot or on-demand). + - Requirements (`requirements`): Set acceptable (`In`) and unacceptable (`Out`) Kubernetes and Karpenter values for node provisioning based on [Well-Known Labels](https://kubernetes.io/docs/reference/labels-annotations-taints/) and [cloud-specific settings]({{}}). These can include [instance types](https://kubernetes.io/docs/reference/labels-annotations-taints/#nodekubernetesioinstance-type), [zones](https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone), [computer architecture](https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetes-io-arch), and [capacity type]({{}}) (such as AWS spot or on-demand). - Limits (`limits`): Lets you set limits on the total CPU and Memory that can be used by the cluster, effectively stopping further node provisioning when those limits have been reached. * **Deploy workloads**: When deploying workloads, you can request that scheduling constraints be met to direct which nodes Karpenter provisions for those workloads. Use any of the following Pod spec constraints when you deploy pods: diff --git a/website/content/en/preview/concepts/_index.md b/website/content/en/preview/concepts/_index.md index 74ec1368b573..90a5aa85f98e 100755 --- a/website/content/en/preview/concepts/_index.md +++ b/website/content/en/preview/concepts/_index.md @@ -44,7 +44,7 @@ Here are some things to know about the Karpenter provisioner: * **Provisioner CR**: Karpenter defines a Custom Resource called a Provisioner to specify provisioning configuration. Each provisioner manages a distinct set of nodes, but pods can be scheduled to any provisioner that supports its scheduling constraints. A provisioner contains constraints that impact the nodes that can be provisioned and attributes of those nodes (such timers for removing nodes). -See [Provisioning]({{}}) docs for a description of settings and provisioner examples. +See [Provisioning]({{}}) docs for a description of settings and provisioner examples. * **Well-known labels**: The provisioner can use well-known Kubernetes labels to allow pods to request only certain instance types, architectures, operating systems, or other attributes when creating nodes. See [Well-Known Labels, Annotations and Taints](https://kubernetes.io/docs/reference/labels-annotations-taints/) for details. diff --git a/website/content/en/preview/concepts/deprovisioning.md b/website/content/en/preview/concepts/disruption.md similarity index 98% rename from website/content/en/preview/concepts/deprovisioning.md rename to website/content/en/preview/concepts/disruption.md index 290ffef8c979..e8f6b7a5dd57 100644 --- a/website/content/en/preview/concepts/deprovisioning.md +++ b/website/content/en/preview/concepts/disruption.md @@ -1,9 +1,9 @@ --- -title: "Deprovisioning" -linkTitle: "Deprovisioning" +title: "Disruption" +linkTitle: "Disruption" weight: 4 description: > - Understand different ways Karpenter deprovisions nodes + Understand different ways Karpenter disrupts nodes --- ## Control Flow @@ -55,7 +55,7 @@ There are both automated and manual ways of deprovisioning nodes provisioned by * **Consolidation**: Karpenter works to actively reduce cluster cost by identifying when: * Nodes can be removed as their workloads will run on other nodes in the cluster. * Nodes can be replaced with cheaper variants due to a change in the workloads. -* **Drift**: Karpenter will annotate nodes as drifted and deprovision nodes that have drifted from their desired specification. See [Drift]({{}}) to see which fields are considered. +* **Drift**: Karpenter will annotate nodes as drifted and deprovision nodes that have drifted from their desired specification. See [Drift]({{}}) to see which fields are considered. * **Interruption**: If enabled, Karpenter will watch for upcoming involuntary interruption events that could affect your nodes (health events, spot interruption, etc.) and will cordon, drain, and terminate the node(s) ahead of the event to reduce workload disruption. {{% alert title="Note" color="primary" %}} diff --git a/website/content/en/preview/concepts/node-templates.md b/website/content/en/preview/concepts/node-templates.md deleted file mode 100644 index 41e924b880b9..000000000000 --- a/website/content/en/preview/concepts/node-templates.md +++ /dev/null @@ -1,693 +0,0 @@ ---- -title: "Node Templates" -linkTitle: "Node Templates" -weight: 2 -description: > - Configure AWS specific settings ---- - -Node Templates enable configuration of AWS specific settings. -Each provisioner must reference an AWSNodeTemplate using `spec.providerRef`. -Multiple provisioners may point to the same AWSNodeTemplate. - -```yaml -apiVersion: karpenter.sh/v1alpha5 -kind: Provisioner -metadata: - name: default -spec: - providerRef: - name: default ---- -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -metadata: - name: default -spec: - subnetSelector: { ... } # required, discovers tagged subnets to attach to instances - securityGroupSelector: { ... } # required, discovers tagged security groups to attach to instances - instanceProfile: "..." # optional, overrides the node's identity from global settings - amiFamily: "..." # optional, resolves a default ami and userdata - amiSelector: { ... } # optional, discovers tagged amis to override the amiFamily's default - userData: "..." # optional, overrides autogenerated userdata with a merge semantic - tags: { ... } # optional, propagates tags to underlying EC2 resources - metadataOptions: { ... } # optional, configures IMDS for the instance - blockDeviceMappings: [ ... ] # optional, configures storage devices for the instance - detailedMonitoring: "..." # optional, configures detailed monitoring for the instance -status: - subnets: { ... } # resolved subnets - securityGroups: { ... } # resolved security groups - amis: { ... } # resolved AMIs -``` -Refer to the [Provisioner docs]({{}}) for settings applicable to all providers. To explore various `AWSNodeTemplate` configurations, refer to the examples provided [in the Karpenter Github repository](https://github.com/aws/karpenter/blob/main/examples/provisioner/). - -See below for other AWS provider-specific parameters. - -## spec.subnetSelector - -The `AWSNodeTemplate` discovers subnets using [AWS tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). -Subnets may be specified by any AWS tag, including `Name`. Selecting tag values using wildcards (`*`) is supported. -Subnet IDs may be specified by using the key `aws-ids` and then passing the IDs as a comma-separated string value. -When launching nodes, a subnet is automatically chosen that matches the desired zone. -If multiple subnets exist for a zone, the one with the most available IP addresses will be used. - -**Examples** - -Select all with a specified tag key: -```yaml -spec: - subnetSelector: - karpenter.sh/discovery/MyClusterName: '*' -``` - -Select by name and tag (all criteria must match):: -```yaml -spec: - subnetSelector: - Name: my-subnet - MyTag: '' # matches all resources with the tag -``` - -Select using comma separated tag values: -```yaml -spec: - subnetSelector: - Name: "my-subnet-1,my-subnet-2" -``` - -Select using wildcards: -```yaml -spec: - subnetSelector: - Name: "*Public*" - -``` - -Select by ID: -```yaml -spec: - subnetSelector: - aws-ids: "subnet-09fa4a0a8f233a921,subnet-0471ca205b8a129ae" -``` - -## spec.securityGroupSelector - -The security group of an instance is comparable to a set of firewall rules. -EKS creates at least two security groups by default, [review the documentation](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html) for more info. -Security groups may be specified by any AWS tag, including "Name". Selecting tags using wildcards (`*`) is supported. - -{{% alert title="Note" color="primary" %}} -When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use the `kubernetes.io/cluster/$CLUSTER_NAME` tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See [this issue](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2367) for more details. -{{% /alert %}} - -To verify if this restriction affects you, run the following commands. -```bash -CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)" - -aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text -``` - -If multiple securityGroups are printed, you will need a more specific securityGroupSelector. We generally recommend that you use the `karpenter.sh/discovery: $CLUSTER_NAME` tag selector instead. - -**Examples** - -Select all assigned to a cluster: -```yaml -spec: - securityGroupSelector: - karpenter.sh/discovery: "${CLUSTER_NAME}" -``` - -Select all with a specified tag key: -```yaml -spec: - securityGroupSelector: - MyTag: '*' -``` - -Select by name and tag (all criteria must match): -```yaml -spec: - securityGroupSelector: - Name: my-security-group - MyTag: '' # matches all resources with the tag -``` - -Select by comma-separated tag values: -```yaml -spec: - securityGroupSelector: - Name: "my-security-group-1,my-security-group-2" -``` - -Select by name using a wildcard: -```yaml -spec: - securityGroupSelector: - Name: "*Public*" -``` - -Select by ID: -```yaml -spec: - securityGroupSelector: - aws-ids: "sg-063d7acfb4b06c82c,sg-06e0cf9c198874591" -``` - -## spec.instanceProfile - -An `InstanceProfile` is a way to pass a single IAM role to EC2 instance launched the provisioner. -A default profile is configured in global settings, but may be overridden here. -The `AWSNodeTemplate` will not create an `InstanceProfile` automatically. -The `InstanceProfile` must refer to a `Role` that has permission to connect to the cluster. -```yaml -spec: - instanceProfile: MyInstanceProfile -``` - -## spec.amiFamily - -The AMI used when provisioning nodes can be controlled by the `amiFamily` field. Based on the value set for `amiFamily`, Karpenter will automatically query for the appropriate [EKS optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-amis.html) via AWS Systems Manager (SSM). When an `amiFamily` of `Custom` is chosen, then an `amiSelector` must be specified that informs Karpenter on which custom AMIs are to be used. - -Currently, Karpenter supports `amiFamily` values `AL2`, `Bottlerocket`, `Ubuntu`, `Windows2019`, `Windows2022` and `Custom`. GPUs are only supported with `AL2` and `Bottlerocket`. The `AL2` amiFamily does not support ARM64 GPU instance types unless you specify a custom amiSelector. - -{{% alert title="Defaults" color="secondary" %}} -If no `amiFamily` is defined, Karpenter will set the default `amiFamily` to AL2 - -```yaml -spec: - amiFamily: AL2 -``` -{{% /alert %}} - -## spec.amiSelector - -AMISelector is used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through `aws::` prefixed filters (`aws::ids`, `aws::owners` and `aws::name`) and [AWS tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). This field is optional, and Karpenter will use the latest EKS-optimized AMIs if an amiSelector is not specified. - -To select an AMI by name, use `aws::name`. EC2 AMIs may be specified by any AWS tag, including `Name`. Selecting by tag or by name using wildcards (`*`) is supported. - -EC2 AMI IDs may be specified by using the key `aws::ids` (`aws-ids` is also supported) and then passing the IDs as a comma-separated string value. - -To ensure that AMIs are owned by the expected owner, use `aws::owners` which expects a comma-separated list of AWS account owners - you can use a combination of account aliases (e.g. `self` `amazon`, `your-aws-account-name`) and account IDs. If this is not set, *and* `aws::ids`/`aws-ids` are not set, it defaults to `self,amazon`. - -{{% alert title="Note" color="primary" %}} -If you use only `aws::owners`, Karpenter will discover all images that are owned by those specified, selecting the most recently created ones to be used. If you specify `aws::owners`, but nothing else, there is a larger chance that Karpenter could select an image that is not compatible with your instance type. To lower this chance, it is recommended to use `aws::name` or `aws::ids` if you're using `aws::owners` to select a subset of images that you have validated are compatible with your selected instance types. -{{% /alert %}} - -### AMI Selection - -If an `amiSelector` matches more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints: - -* When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType's requirements. -* If multiple AMIs are found that can be used, Karpenter will choose the latest one. -* If no AMIs are found that can be used, then no nodes will be provisioned. - -If you need to express other constraints for an AMI beyond architecture, you can express these constraints as tags on the AMI. For example, if you want to limit an EC2 AMI to only be used with instanceTypes that have an `nvidia` GPU, you can specify an EC2 tag with a key of `karpenter.k8s.aws/instance-gpu-manufacturer` and value `nvidia` on that AMI. - -All labels defined [in the scheduling documentation](../scheduling#well-known-labels) can be used as requirements for an EC2 AMI. - -```bash -> aws ec2 describe-images --image-id ami-123 --query Images[0].Tags -[ - { - "Key": "karpenter.sh/discovery", - "Value": "my-cluster" - }, - { - "Key": "Name", - "Value": "amazon-eks-node-1.21-customized-v0" - }, - { - "Key": "karpenter.k8s.aws/instance-gpu-manufacturer", - "Value": "nvidia" - } -] -``` - -#### Examples - -Select all AMIs with a specified tag: -```yaml - amiSelector: - karpenter.sh/discovery/MyClusterName: '*' -``` - -Select AMIs by the AMI name: -```yaml - amiSelector: - aws::name: my-ami -``` -Select AMIs by the Name tag: -```yaml - amiSelector: - Name: my-ami -``` - -Select AMIs by name and a specific owner: -```yaml - amiSelector: - aws::name: my-ami - aws::owners: self/ownerAccountID -``` - -Select AMIs by an arbitrary AWS tag key/value pair: -```yaml - amiSelector: - MyAMITag: value -``` - -Specify AMIs explicitly by ID: -```yaml - amiSelector: - aws::ids: "ami-123,ami-456" -``` - -## spec.tags - -Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of AWS tags are listed below. - -``` -Name: karpenter.sh/provisioner-name/ -karpenter.sh/provisioner-name: -kubernetes.io/cluster/: owned -``` - -Additional tags can be added in the AWSNodeTemplate tags section which are merged with global tags in `aws.tags` (located in karpenter-global-settings ConfigMap). -```yaml -spec: - tags: - InternalAccountingTag: 1234 - dev.corp.net/app: Calculator - dev.corp.net/team: MyTeam -``` - -Karpenter allows overrides of the default "Name" tag but does not allow overrides to restricted domains (such as "karpenter.sh", "karpenter.k8s.aws", and "kubernetes.io/cluster"). This ensures that Karpenter is able to correctly auto-discover machines that it owns. - -## spec.metadataOptions - -Control the exposure of [Instance Metadata Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) on EC2 Instances launched by this provisioner using a generated launch template. - -Refer to [recommended, security best practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node) for limiting exposure of Instance Metadata and User Data to pods. - -If metadataOptions are omitted from this provisioner, the following default settings will be used. - -```yaml -spec: - metadataOptions: - httpEndpoint: enabled - httpProtocolIPv6: disabled - httpPutResponseHopLimit: 2 - httpTokens: required -``` - -## spec.blockDeviceMappings - -The `blockDeviceMappings` field in an AWSNodeTemplate can be used to control the Elastic Block Storage (EBS) volumes that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMI Family specified. For example, the `Bottlerocket` AMI Family defaults with two block device mappings, one for Bottlerocket's control volume and the other for container resources such as images and logs. - -Learn more about [block device mappings](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html). - -### Examples - -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -spec: - blockDeviceMappings: - - deviceName: /dev/xvda - ebs: - volumeSize: 100Gi - volumeType: gp3 - iops: 10000 - encrypted: true - kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" - deleteOnTermination: true - throughput: 125 - snapshotID: snap-0123456789 -``` - -{{% alert title="Defaults" color="secondary" %}} -If no `blockDeviceMappings` is defined, Karpenter will set the default `blockDeviceMappings` to the following for the given AMI family. - -#### AL2 -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -spec: - blockDeviceMappings: - - deviceName: /dev/xvda - ebs: - volumeSize: 20Gi - volumeType: gp3 - encrypted: true -``` - -#### Bottlerocket -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -spec: - blockDeviceMappings: - # Root device - - deviceName: /dev/xvda - ebs: - volumeSize: 4Gi - volumeType: gp3 - encrypted: true - # Data device: Container resources such as images and logs - - deviceName: /dev/xvdb - ebs: - volumeSize: 20Gi - volumeType: gp3 - encrypted: true -``` - -#### Ubuntu -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -spec: - blockDeviceMappings: - - deviceName: /dev/sda1 - ebs: - volumeSize: 20Gi - volumeType: gp3 - encrypted: true -``` - -#### Windows2019, Windows2022 -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -spec: - blockDeviceMappings: - - deviceName: /dev/sda1 - ebs: - volumeSize: 50Gi - volumeType: gp3 - encrypted: true -``` -{{% /alert %}} - -## spec.userData - -You can control the UserData that is applied to your worker nodes via this field. - -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -metadata: - name: bottlerocket-example -spec: - amiFamily: Bottlerocket - instanceProfile: MyInstanceProfile - subnetSelector: - karpenter.sh/discovery: my-cluster - securityGroupSelector: - karpenter.sh/discovery: my-cluster - userData: | - [settings.kubernetes] - "kube-api-qps" = 30 - "shutdown-grace-period" = "30s" - "shutdown-grace-period-for-critical-pods" = "30s" - [settings.kubernetes.eviction-hard] - "memory.available" = "20%" - amiSelector: - karpenter.sh/discovery: my-cluster -``` - -This example adds SSH keys to allow remote login to the node (replace *my-authorized_keys* with your key file): - -{{% alert title="Note" color="primary" %}} -Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. -See [Node NotReady]({{< ref "../troubleshooting/#node-notready" >}}) troubleshooting for an example of starting an SSM session from the command line or [EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html) documentation to connect to nodes using SSH. -{{% /alert %}} - -```yaml -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -metadata: - name: al2-example -spec: - amiFamily: AL2 - instanceProfile: MyInstanceProfile - subnetSelector: - karpenter.sh/discovery: my-cluster - securityGroupSelector: - karpenter.sh/discovery: my-cluster - userData: | - #!/bin/bash - mkdir -p ~ec2-user/.ssh/ - touch ~ec2-user/.ssh/authorized_keys - cat >> ~ec2-user/.ssh/authorized_keys < >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 -/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ ---use-max-pods false \ ---container-runtime containerd \ ---kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test --max-pods=110' ---//-- -``` - -You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet: - -``` -apiVersion: karpenter.k8s.aws/v1alpha1 -kind: AWSNodeTemplate -metadata: - name: kubelet-config-example -spec: - subnetSelector: - karpenter.sh/discovery: my-cluster - securityGroupSelector: - karpenter.sh/discovery: my-cluster - userData: | - #!/bin/bash - echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json -``` - -#### Windows - -* Your UserData must be specified as PowerShell commands. -* The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet. -* Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before. - -Consider the following example to understand how your custom UserData settings will be merged in. - -Your UserData - -``` -Write-Host "Running custom user data script" -``` - -Final merged UserData - -``` - -Write-Host "Running custom user data script" -[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" -& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/provisioner-name=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10' - -``` - -{{% alert title="Windows Support Notice" color="warning" %}} -Currently, Karpenter does not specify `-ServiceCIDR` to [EKS Windows AMI Bootstrap script](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html#bootstrap-script-configuration-parameters). -Windows worker nodes will use `172.20.0.0/16` or `10.100.0.0/16` for Kubernetes service IP address ranges based on the IP address of the primary interface. -The effective ServiceCIDR can be verified at `$env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf` on the worker node. - -Support for the Windows ServiceCIDR argument can be tracked in a [Karpenter Github Issue](https://github.com/aws/karpenter/issues/4088). Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround. - -```yaml -spec: - userData: | - $global:EKSCluster = Get-EKSCluster -Name my-cluster -``` -{{% /alert %}} - -## spec.detailedMonitoring - -Enabling detailed monitoring on the node template controls the [EC2 detailed monitoring](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches. -```yaml -spec: - detailedMonitoring: true -``` - -## status.subnets -`status.subnets` contains the `id` and `zone` of the subnets utilized during node launch. The subnets are sorted by the available IP address count in decreasing order. - -**Examples** - -```yaml -status: - subnets: - - id: subnet-0a462d98193ff9fac - zone: us-east-2b - - id: subnet-0322dfafd76a609b6 - zone: us-east-2c - - id: subnet-0727ef01daf4ac9fe - zone: us-east-2b - - id: subnet-00c99aeafe2a70304 - zone: us-east-2a - - id: subnet-023b232fd5eb0028e - zone: us-east-2c - - id: subnet-03941e7ad6afeaa72 - zone: us-east-2a -``` - -## status.securityGroups -`status.securityGroups` contains the `id` and `name` of the security groups utilized during node launch. - -**Examples** - -```yaml - status: - securityGroups: - - id: sg-041513b454818610b - name: ClusterSharedNodeSecurityGroup - - id: sg-0286715698b894bca - name: ControlPlaneSecurityGroup-1AQ073TSAAPW -``` - -## status.amis -`status.amis` contains the `id`, `name`, and `requirements` of the amis utilized during node launch. - -**Examples** - -```yaml - amis: - - id: ami-03c3a3dcda64f5b75 - name: amazon-linux-2-gpu - requirements: - - key: kubernetes.io/arch - operator: In - values: - - amd64 - - key: karpenter.k8s.aws/instance-accelerator-manufacturer - operator: In - values: - - aws - - nvidia - - id: ami-06afb2d101cc4b8bd - name: amazon-linux-2-arm64 - requirements: - - key: kubernetes.io/arch - operator: In - values: - - arm64 - - key: karpenter.k8s.aws/instance-accelerator-manufacturer - operator: NotIn - values: - - aws - - nvidia - - id: ami-0e28b76d768af234e - name: amazon-linux-2 - requirements: - - key: kubernetes.io/arch - operator: In - values: - - amd64 - - key: karpenter.k8s.aws/instance-accelerator-manufacturer - operator: NotIn - values: - - aws - - nvidia -``` \ No newline at end of file diff --git a/website/content/en/preview/concepts/nodeclasses.md b/website/content/en/preview/concepts/nodeclasses.md new file mode 100644 index 000000000000..ca668aef11f0 --- /dev/null +++ b/website/content/en/preview/concepts/nodeclasses.md @@ -0,0 +1,883 @@ +--- +title: "NodeClasses" +linkTitle: "NodeClasses" +weight: 2 +description: > + Configure AWS-specific settings with EC2NodeClasses +--- + +Node Classes enable configuration of AWS specific settings. +Each NodePool must reference an EC2NodeClass using `spec.template.spec.nodeClassRef`. +Multiple NodePools may point to the same EC2NodeClass. + +```yaml +apiVersion: karpenter.sh/v1beta1 +kind: NodePool +metadata: + name: default +spec: + template: + spec: + nodeClassRef: + apiVersion: karpenter.k8s.aws/v1beta1 + kind: EC2NodeClass + name: default +--- +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +metadata: + name: default +spec: + # required, resolves a default ami and userdata + amiFamily: AL2 + + # required, discovers subnets to attach to instances + subnetSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" + + # required, discovers security groups to attach to instances + securityGroupSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" + + # optional, discovers amis to override the amiFamily's default + amiSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" + + # optional, IAM role to use for the node identity + role: "KarpenterNodeRole-${CLUSTER_NAME}" + + # optional, overrides autogenerated userdata with a merge semantic + userData: | + echo "Hello world" + + # optional, propagates tags to underlying EC2 resources + tags: + team: team-a + app: team-a-app + + # optional, configures IMDS for the instance + metadataOptions: + httpEndpoint: enabled + httpProtocolIPv6: disabled + httpPutResponseHopLimit: 2 + httpTokens: required + + # optional, configures storage devices for the instance + blockDeviceMappings: + - deviceName: /dev/xvda + ebs: + volumeSize: 100Gi + volumeType: gp3 + iops: 10000 + encrypted: true + kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" + deleteOnTermination: true + throughput: 125 + snapshotID: snap-0123456789 + + # optional, configures detailed monitoring for the instance + detailedMonitoring: true +status: + # resolved subnets + subnets: + - id: subnet-0a462d98193ff9fac + zone: us-east-2b + - id: subnet-0322dfafd76a609b6 + zone: us-east-2c + - id: subnet-0727ef01daf4ac9fe + zone: us-east-2b + - id: subnet-00c99aeafe2a70304 + zone: us-east-2a + - id: subnet-023b232fd5eb0028e + zone: us-east-2c + - id: subnet-03941e7ad6afeaa72 + zone: us-east-2a + + # resolved security groups + securityGroups: + - id: sg-041513b454818610b + name: ClusterSharedNodeSecurityGroup + - id: sg-0286715698b894bca + name: ControlPlaneSecurityGroup-1AQ073TSAAPW + + # resolved AMIs + amis: + - id: ami-01234567890123456 + name: custom-ami-amd64 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - amd64 + - id: ami-01234567890123456 + name: custom-ami-arm64 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - arm64 + + # generated instance profile name + instanceProfile: "${CLUSTER_NAME}-0123456778901234567789" +``` +Refer to the [NodePool docs]({{}}) for settings applicable to all providers. To explore various `EC2NodeClass` configurations, refer to the examples provided [in the Karpenter Github repository](https://github.com/aws/karpenter/blob/main/examples/nodepool/). + +## spec.amiFamily + +AMIFamily is a required field, dictating both the default bootstrapping logic for nodes provisioned through this `EC2NodeClass` but also selecting a group of recommended, latest AMIs by default. Currently, Karpenter supports `amiFamily` values `AL2`, `Bottlerocket`, `Ubuntu`, `Windows2019`, `Windows2022` and `Custom`. GPUs are only supported by default with `AL2` and `Bottlerocket`. The `AL2` amiFamily does not support ARM64 GPU instance types unless you specify a custom amiSelector. Default bootstrapping logic is shown below for each of the supported families. + +### AL2 + +```console +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="//" + +--// +Content-Type: text/x-shellscript; charset="us-ascii" + +#!/bin/bash -xe +exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 +/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ +--dns-cluster-ip '10.100.0.10' \ +--use-max-pods false \ +--container-runtime containerd \ +--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test --max-pods=110' +--//-- +``` + +### Bottlerocket + +```toml +[settings] +[settings.kubernetes] +api-server = 'https://test-cluster' +cluster-certificate = 'ca-bundle' +cluster-name = 'test-cluster' +cluster-dns-ip = '10.100.0.10' +max-pods = 110 + +[settings.kubernetes.node-labels] +'karpenter.sh/capacity-type' = 'on-demand' +'karpenter.sh/provisioner-name' = 'test' +``` + +### Ubuntu + +```console +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="//" + +--// +Content-Type: text/x-shellscript; charset="us-ascii" + +#!/bin/bash -xe +exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 +/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ +--dns-cluster-ip '10.100.0.10' \ +--use-max-pods false \ +--kubelet-extra-args '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test" --max-pods=110' +--//-- +``` + +### Windows2019 + +```powershell + +[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" +& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test" --max-pods=110' -DNSClusterIP '10.100.0.10' + +``` + +### Windows2022 + +```powershell + +[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" +& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test" --max-pods=110' -DNSClusterIP '10.100.0.10' + +``` + +{{% alert title="Note" color="primary" %}} +Karpenter will automatically query for the appropriate [EKS optimized AMI](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-amis.html) via AWS Systems Manager (SSM). In the case of the `Custom` AMIFamily, no default AMIs are defined. As a result, `amiSelectorTerms` must be specified to inform Karpenter on which custom AMIs are to be used. +{{% /alert %}} + +### Custom + +The `Custom` AMIFamily ships without any default userData to allow you to configure custom bootstrapping for control planes or images that don't support the default methods from the other families. + +## spec.subnetSelectorTerms + +The `EC2NodeClass` discovers subnets through ids or [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). When launching nodes, a subnet is automatically chosen that matches the desired zone. If multiple subnets exist for a zone, the one with the most available IP addresses will be used. + +{{% alert title="Tip" color="secondary" %}} +Subnets may be specified by any tag, including `Name`. Selecting tag values using wildcards (`*`) is supported. +{{% /alert %}} + +#### Examples + +Select all with a specified tag key: +```yaml +spec: + subnetSelectorTerms: + - tags: + karpenter.sh/discovery/MyClusterName: '*' +``` + +Select by name and tag (all criteria must match): +```yaml +spec: + subnetSelectorTerms: + - tags: + Name: my-subnet + MyTag: '' # matches all resources with the tag +``` + +Select using multiple tag terms: +```yaml +spec: + subnetSelectorTerms: + - tags: + Name: "my-subnet-1" + - tags: + Name: "my-subnet-2" +``` + +Select using wildcards: +```yaml +spec: + subnetSelectorTerms: + - tags: + Name: "*Public*" + +``` + +Select using ids: +```yaml +spec: + subnetSelectorTerms: + - id: "subnet-09fa4a0a8f233a921" + - id: "subnet-0471ca205b8a129ae" +``` + +## spec.securityGroupSelectorTerms + +The security group of an instance is comparable to a set of firewall rules. +[EKS creates at least two security groups by default](https://docs.aws.amazon.com/eks/latest/userguide/sec-group-reqs.html). + +{{% alert title="Tip" color="secondary" %}} +Security groups may be specified by any tag, including "Name". Selecting tags using wildcards (`*`) is supported. +{{% /alert %}} + +{{% alert title="Note" color="primary" %}} +When launching nodes, Karpenter uses all the security groups that match the selector. If you choose to use the `kubernetes.io/cluster/$CLUSTER_NAME` tag for discovery, note that this may result in failures using the AWS Load Balancer controller. The Load Balancer controller only supports a single security group having that tag key. See [this issue](https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2367) for more details. + +To verify if this restriction affects you, run the following commands. +```bash +CLUSTER_VPC_ID="$(aws eks describe-cluster --name $CLUSTER_NAME --query cluster.resourcesVpcConfig.vpcId --output text)" + +aws ec2 describe-security-groups --filters Name=vpc-id,Values=$CLUSTER_VPC_ID Name=tag-key,Values=kubernetes.io/cluster/$CLUSTER_NAME --query 'SecurityGroups[].[GroupName]' --output text +``` + +If multiple securityGroups are printed, you will need a more specific securityGroupSelector. We generally recommend that you use the `karpenter.sh/discovery: $CLUSTER_NAME` tag selector instead. +{{% /alert %}} + +#### Examples + +Select all assigned to a cluster: +```yaml +spec: + securityGroupSelectorTerms: + - tags: + kubernetes.io/cluster/$CLUSTER_NAME: "owned" +``` + +Select all with a specified tag key: +```yaml +spec: + securityGroupSelectorTerms: + - tags: + MyTag: '*' +``` + +Select by name and tag (all criteria must match): +```yaml +spec: + securityGroupSelectorTerms: + - name: my-security-group + tags: + MyTag: '*' # matches all resources with the tag +``` + +Select using multiple tag terms: +```yaml +spec: + securityGroupSelectorTerms: + - tags: + Name: "my-security-group-1" + - tags: + Name: "my-security-group-2" +``` + +Select by name using a wildcard: +```yaml +spec: + securityGroupSelectorTerms: + - name: "*Public*" +``` + +Select using ids: +```yaml +spec: + securityGroupSelector: + - id: "sg-063d7acfb4b06c82c" + - id: "sg-06e0cf9c198874591" +``` + +## spec.amiSelectorTerms + +AMISelector is used to configure custom AMIs for Karpenter to use, where the AMIs are discovered through ids, owners, name, and [tags](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html). This field is optional, and Karpenter will use the latest EKS-optimized AMIs for the AMIFamily if no amiSelectorTerms are specified. To select an AMI by name, use the `name` field in the selector term. To select an AMI by id, use the `id` field in the selector term. To ensure that AMIs are owned by the expected owner, use the `owner` field - you can use a combination of account aliases (e.g. `self` `amazon`, `your-aws-account-name`) and account IDs. If this is not set, it defaults to `self,amazon`. + +{{% alert title="Tip" color="secondary" %}} +AMIs may be specified by any AWS tag, including `Name`. Selecting by tag or by name using wildcards (`*`) is supported. +{{% /alert %}} + +{{% alert title="Note" color="primary" %}} +If `amiSelectorTerms` match more than one AMI, Karpenter will automatically determine which AMI best fits the workloads on the launched worker node under the following constraints: + +* When launching nodes, Karpenter automatically determines which architecture a custom AMI is compatible with and will use images that match an instanceType's requirements. +* If multiple AMIs are found that can be used, Karpenter will choose the latest one. +* If no AMIs are found that can be used, then no nodes will be provisioned. +{{% /alert %}} + +#### Examples + +Select all with a specified tag: +```yaml + amiSelectorTerms: + - tags: + karpenter.sh/discovery/MyClusterName: '*' +``` + +Select by name: +```yaml + amiSelectorTerms: + - name: my-ami +``` + +Select by `Name` tag: +```yaml + amiSelector: + - tags: + Name: my-ami +``` + +Select by name and owner: +```yaml + amiSelector: + - name: my-ami + owner: self + - name: my-ami + owner: 0123456789 +``` + +Select by name using a wildcard: +```yaml +spec: + amiSelectorTerms: + - name: "*EKS*" +``` + +Select by all under an owner: +```yaml +spec: + amiSelectorTerms: + - name: "*" + owner: self +``` + +Specify using ids: +```yaml + amiSelector: + - id: "ami-123" + - id: "ami-456" +``` + +## spec.role + +`Role` is a required field and is necessary to tell Karpenter which identity nodes from this `EC2NodeClass` should assume. If using the [Karpenter Getting Started Guide]({{}}) to deploy Karpenter, you can use the `KarpenterNodeRole-$CLUSTER_NAME` role provisioned by that process. + +```yaml +spec: + role: "KarpenterNodeRole-$CLUSTER_NAME" +``` + +## spec.tags + +Karpenter adds tags to all resources it creates, including EC2 Instances, EBS volumes, and Launch Templates. The default set of tags are listed below. + +```yaml +Name: +karpenter.sh/nodeclaim: +karpenter.sh/nodepool: +kubernetes.io/cluster/: owned +``` + +Additional tags can be added in the tags section, which will be merged with the default tags specified above. +```yaml +spec: + tags: + InternalAccountingTag: 1234 + dev.corp.net/app: Calculator + dev.corp.net/team: MyTeam +``` + +{{% alert title="Note" color="primary" %}} +Karpenter allows overrides of the default "Name" tag but does not allow overrides to restricted domains (such as "karpenter.sh", "karpenter.k8s.aws", and "kubernetes.io/cluster"). This ensures that Karpenter is able to correctly auto-discover nodes that it owns. +{{% /alert %}} + +## spec.metadataOptions + +Control the exposure of [Instance Metadata Service](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) on EC2 Instances launched by this NodePool using a generated launch template. + +Refer to [recommended, security best practices](https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node) for limiting exposure of Instance Metadata and User Data to pods. + +If metadataOptions are omitted from this provisioner, the following default settings are applied to the `EC2NodeClass`. + +```yaml +spec: + metadataOptions: + httpEndpoint: enabled + httpProtocolIPv6: disabled + httpPutResponseHopLimit: 2 + httpTokens: required +``` + +## spec.blockDeviceMappings + +The `blockDeviceMappings` field in an `EC2NodeClass` can be used to control the [Elastic Block Storage (EBS) volumes](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#instance-block-device-mapping) that Karpenter attaches to provisioned nodes. Karpenter uses default block device mappings for the AMIFamily specified. For example, the `Bottlerocket` AMI Family defaults with two block device mappings, one for Bottlerocket's control volume and the other for container resources such as images and logs. + +```yaml +spec: + blockDeviceMappings: + - deviceName: /dev/xvda + ebs: + volumeSize: 100Gi + volumeType: gp3 + iops: 10000 + encrypted: true + kmsKeyID: "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab" + deleteOnTermination: true + throughput: 125 + snapshotID: snap-0123456789 +``` + +The following blockDeviceMapping defaults are used for each `AMIFamily` if no `blockDeviceMapping` overrides are specified in the `EC2NodeClass` + +### AL2 +```yaml +spec: + blockDeviceMappings: + - deviceName: /dev/xvda + ebs: + volumeSize: 20Gi + volumeType: gp3 + encrypted: true +``` + +### Bottlerocket +```yaml +spec: + blockDeviceMappings: + # Root device + - deviceName: /dev/xvda + ebs: + volumeSize: 4Gi + volumeType: gp3 + encrypted: true + # Data device: Container resources such as images and logs + - deviceName: /dev/xvdb + ebs: + volumeSize: 20Gi + volumeType: gp3 + encrypted: true +``` + +### Ubuntu +```yaml +spec: + blockDeviceMappings: + - deviceName: /dev/sda1 + ebs: + volumeSize: 20Gi + volumeType: gp3 + encrypted: true +``` + +### Windows2019/Windows2022 +```yaml +spec: + blockDeviceMappings: + - deviceName: /dev/sda1 + ebs: + volumeSize: 50Gi + volumeType: gp3 + encrypted: true +``` + +### Custom + +The `Custom` AMIFamily ships without any default `blockDeviceMappings`. + +## spec.userData + +You can control the UserData that is applied to your worker nodes via this field. This allows you to run custom scripts or pass-through custom configuration to Karpenter instances on start-up. + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +metadata: + name: bottlerocket-example +spec: + ... + amiFamily: Bottlerocket + userData: | + [settings.kubernetes] + "kube-api-qps" = 30 + "shutdown-grace-period" = "30s" + "shutdown-grace-period-for-critical-pods" = "30s" + [settings.kubernetes.eviction-hard] + "memory.available" = "20%" +``` + +This example adds SSH keys to allow remote login to the node (replace *my-authorized_keys* with your key file): + +{{% alert title="Note" color="primary" %}} +Instead of using SSH as set up in this example, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell access to Karpenter nodes. +See [Node NotReady]({{< ref "../troubleshooting/#node-notready" >}}) troubleshooting for an example of starting an SSM session from the command line or [EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html) documentation to connect to nodes using SSH. +{{% /alert %}} + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +metadata: + name: al2-example +spec: + ... + amiFamily: AL2 + userData: | + #!/bin/bash + mkdir -p ~ec2-user/.ssh/ + touch ~ec2-user/.ssh/authorized_keys + cat >> ~ec2-user/.ssh/authorized_keys <}}) section for more details on these defaults. View the sections below to understand the different merge strategies for each AMIFamily. + +### AL2/Ubuntu + +* Your UserData can be in the [MIME multi part archive](https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive) format. +* Karpenter will transform your custom user-data as a MIME part, if necessary, and then merge a final MIME part to the end of your UserData parts which will bootstrap the worker node. Karpenter will have full control over all the parameters being passed to the bootstrap script. + * Karpenter will continue to set MaxPods, ClusterDNS and all other parameters defined in `spec.kubeletConfiguration` as before. + +Consider the following example to understand how your custom UserData will be merged - + +#### Passed-in UserData (bash) + +```bash +#!/bin/bash +echo "Running custom user data script" +``` + +#### Passed-in UserData (MIME) + +```bash +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="BOUNDARY" + +--BOUNDARY +Content-Type: text/x-shellscript; charset="us-ascii" + +#!/bin/bash +echo "Running custom user data script" + +--BOUNDARY-- +``` + +#### Merged UserData + +```bash +MIME-Version: 1.0 +Content-Type: multipart/mixed; boundary="//" + +--// +Content-Type: text/x-shellscript; charset="us-ascii" + +#!/bin/bash +echo "Running custom user data script" + +--// +Content-Type: text/x-shellscript; charset="us-ascii" + +#!/bin/bash -xe +exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1 +/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \ +--use-max-pods false \ +--container-runtime containerd \ +--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/provisioner-name=test --max-pods=110' +--//-- +``` + +{{% alert title="Note" color="primary" %}} +You can also set kubelet-config properties by modifying the kubelet-config.json file before the EKS bootstrap script starts the kubelet: + +```yaml +apiVersion: karpenter.k8s.aws/v1beta1 +kind: EC2NodeClass +metadata: + name: kubelet-config-example +spec: + ... + amiFamily: AL2 + userData: | + #!/bin/bash + echo "$(jq '.kubeAPIQPS=50' /etc/kubernetes/kubelet/kubelet-config.json)" > /etc/kubernetes/kubelet/kubelet-config.json +``` +{{% /alert %}} + +### Bottlerocket + +* Your UserData must be valid TOML. +* Karpenter will automatically merge settings to ensure successful bootstrap including `cluster-name`, `api-server` and `cluster-certificate`. Any labels and taints that need to be set based on pod requirements will also be specified in the final merged UserData. + * All Kubelet settings that Karpenter applies will override the corresponding settings in the provided UserData. For example, if you've specified `settings.kubernetes.cluster-name`, it will be overridden. + * If MaxPods is specified via the binary arg to Karpenter, the value will override anything specified in the UserData. + * If ClusterDNS is specified via `spec.kubeletConfiguration`, then that value will override anything specified in the UserData. +* Unknown TOML fields will be ignored when the final merged UserData is generated by Karpenter. + +Consider the following example to understand how your custom UserData settings will be merged in. + +#### Passed-in UserData + +```toml +[settings.kubernetes.eviction-hard] +"memory.available" = "12%" +[settings.kubernetes] +"unknown-setting" = "unknown" +[settings.kubernetes.node-labels] +'field.controlled.by/karpenter' = 'will-be-overridden' +``` + +#### Merged UserData + +```toml +[settings] +[settings.kubernetes] +api-server = 'https://cluster' +cluster-certificate = 'ca-bundle' +cluster-name = 'cluster' + +[settings.kubernetes.node-labels] +'karpenter.sh/capacity-type' = 'on-demand' +'karpenter.sh/provisioner-name' = 'provisioner' + +[settings.kubernetes.node-taints] + +[settings.kubernetes.eviction-hard] +'memory.available' = '12%%' +``` + +### Windows2019/Windows2022 + +* Your UserData must be specified as PowerShell commands. +* The UserData specified will be prepended to a Karpenter managed section that will bootstrap the kubelet. +* Karpenter will continue to set ClusterDNS and all other parameters defined in spec.kubeletConfiguration as before. + +Consider the following example to understand how your custom UserData settings will be merged in. + +#### Passed-in UserData + +```powershell +Write-Host "Running custom user data script" +``` + +#### Merged UserData + +```powershell + +Write-Host "Running custom user data script" +[string]$EKSBootstrapScriptFile = "$env:ProgramFiles\Amazon\EKS\Start-EKSBootstrap.ps1" +& $EKSBootstrapScriptFile -EKSClusterName 'test-cluster' -APIServerEndpoint 'https://test-cluster' -Base64ClusterCA 'ca-bundle' -KubeletExtraArgs '--node-labels="karpenter.sh/capacity-type=spot,karpenter.sh/provisioner-name=windows2022" --max-pods=110' -DNSClusterIP '10.0.100.10' + +``` + +{{% alert title="Windows Support Notice" color="warning" %}} +Currently, Karpenter does not specify `-ServiceCIDR` to [EKS Windows AMI Bootstrap script](https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-windows-ami.html#bootstrap-script-configuration-parameters). +Windows worker nodes will use `172.20.0.0/16` or `10.100.0.0/16` for Kubernetes service IP address ranges based on the IP address of the primary interface. +The effective ServiceCIDR can be verified at `$env:ProgramData\Amazon\EKS\cni\config\vpc-bridge.conf` on the worker node. + +Support for the Windows ServiceCIDR argument can be tracked in a [Karpenter Github Issue](https://github.com/aws/karpenter/issues/4088). Currently, if the effective ServiceCIDR is incorrect for your windows worker nodes, you can add the following userData as a workaround. + +```yaml +spec: + userData: | + $global:EKSCluster = Get-EKSCluster -Name my-cluster +``` +{{% /alert %}} + +### Custom + +* No merging is performed, your UserData must perform all setup required of the node to allow it to join the cluster. + +## spec.detailedMonitoring + +Enabling detailed monitoring controls the [EC2 detailed monitoring](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-cloudwatch-new.html) feature. If you enable this option, the Amazon EC2 console displays monitoring graphs with a 1-minute period for the instances that Karpenter launches. + +```yaml +spec: + detailedMonitoring: true +``` + +## status.subnets +[`status.subnets`]({{< ref "#statussubnets" >}}) contains the resolved `id` and `zone` of the subnets that were selected by the [`spec.subnetSelectorTerms`]({{< ref "#specsubnetselectorterms" >}}) for the node class. The subnets will be sorted by the available IP address count in decreasing order. + +#### Examples + +```yaml +spec: + subnetSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" +status: + subnets: + - id: subnet-0a462d98193ff9fac + zone: us-east-2b + - id: subnet-0322dfafd76a609b6 + zone: us-east-2c + - id: subnet-0727ef01daf4ac9fe + zone: us-east-2b + - id: subnet-00c99aeafe2a70304 + zone: us-east-2a + - id: subnet-023b232fd5eb0028e + zone: us-east-2c + - id: subnet-03941e7ad6afeaa72 + zone: us-east-2a +``` + +## status.securityGroups + +[`status.securityGroups`]({{< ref "#statussecuritygroups" >}}) contains the resolved `id` and `name` of the security groups that were selected by the [`spec.securityGroupSelectorTerms`]({{< ref "#specsecuritygroupselectorterms" >}}) for the node class. The subnets will be sorted by the available IP address count in decreasing order. + +#### Examples + +```yaml +spec: + securityGroupSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" +status: + securityGroups: + - id: sg-041513b454818610b + name: ClusterSharedNodeSecurityGroup + - id: sg-0286715698b894bca + name: ControlPlaneSecurityGroup-1AQ073TSAAPW +``` + +## status.amis + +[`status.amis`]({{< ref "#statusamis" >}}) contains the resolved `id`, `name`, and `requirements` of either the default AMIs for the [`spec.amiFamily`]({{< ref "#specamifamily" >}}) or the AMIs selected by the [`spec.amiSelectorTerms`]({{< ref "#specamiselectorterms" >}}) if this field is specified. + +#### Examples + +Default AMIs resolved from the AL2 AMIFamily: + +```yaml +spec: + amiFamily: AL2 +status: + amis: + - id: ami-03c3a3dcda64f5b75 + name: amazon-linux-2-gpu + requirements: + - key: kubernetes.io/arch + operator: In + values: + - amd64 + - key: karpenter.k8s.aws/instance-gpu-count + operator: Exists + - id: ami-03c3a3dcda64f5b75 + name: amazon-linux-2-gpu + requirements: + - key: kubernetes.io/arch + operator: In + values: + - amd64 + - key: karpenter.k8s.aws/instance-accelerator-count + operator: Exists + - id: ami-06afb2d101cc4b8bd + name: amazon-linux-2-arm64 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - arm64 + - key: karpenter.k8s.aws/instance-gpu-count + operator: DoesNotExist + - key: karpenter.k8s.aws/instance-accelerator-count + operator: DoesNotExist + - id: ami-0e28b76d768af234e + name: amazon-linux-2 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - amd64 + - key: karpenter.k8s.aws/instance-gpu-count + operator: DoesNotExist + - key: karpenter.k8s.aws/instance-accelerator-count + operator: DoesNotExist +``` + +AMIs resolved from [`spec.amiSelectorTerms`]({{< ref "#specamiselectorterms" >}}): + +```yaml +spec: + amiFamily: AL2 + amiSelectorTerms: + - tags: + karpenter.sh/discovery: "${CLUSTER_NAME}" +status: + amis: + - id: ami-01234567890123456 + name: custom-ami-amd64 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - amd64 + - id: ami-01234567890123456 + name: custom-ami-arm64 + requirements: + - key: kubernetes.io/arch + operator: In + values: + - arm64 +``` + +## status.instanceProfile + +[`status.instanceProfile`]({{< ref "#statusinstanceprofile" >}}) contains the resolved instance profile generated by Karpenter from the [`spec.role`]({{< ref "#specrole" >}}) + +```yaml +spec: + role: "KarpenterNodeRole-${CLUSTER_NAME}" +status: + instanceProfile: "${CLUSTER_NAME}-0123456778901234567789" +``` \ No newline at end of file diff --git a/website/content/en/preview/concepts/provisioners.md b/website/content/en/preview/concepts/nodepools.md similarity index 99% rename from website/content/en/preview/concepts/provisioners.md rename to website/content/en/preview/concepts/nodepools.md index a65c84eee24e..73a0704d227a 100644 --- a/website/content/en/preview/concepts/provisioners.md +++ b/website/content/en/preview/concepts/nodepools.md @@ -1,9 +1,9 @@ --- -title: "Provisioners" -linkTitle: "Provisioners" +title: "NodePools" +linkTitle: "NodePools" weight: 1 description: > - Learn about Karpenter Provisioners + Configure Karpenter with NodePools --- When you first installed Karpenter, you set up a default Provisioner. @@ -437,7 +437,7 @@ This field points to the cloud provider-specific custom resource. Learn more abo ## spec.consolidation -You can configure Karpenter to deprovision instances through your Provisioner in multiple ways. You can use `spec.ttlSecondsAfterEmpty`, `spec.ttlSecondsUntilExpired` or `spec.consolidation.enabled`. Read [Deprovisioning](../deprovisioning/) for more. +You can configure Karpenter to deprovision instances through your Provisioner in multiple ways. You can use `spec.ttlSecondsAfterEmpty`, `spec.ttlSecondsUntilExpired` or `spec.consolidation.enabled`. Read [Deprovisioning]({{}}) for more. ## Example Use-Cases diff --git a/website/content/en/preview/concepts/scheduling.md b/website/content/en/preview/concepts/scheduling.md index c78fef059fbd..df0b6a7babc9 100755 --- a/website/content/en/preview/concepts/scheduling.md +++ b/website/content/en/preview/concepts/scheduling.md @@ -62,7 +62,7 @@ Its limits are set to 256MiB of memory and 1 CPU. Instance type selection math only uses `requests`, but `limits` may be configured to enable resource oversubscription. -See [Managing Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for details on resource types supported by Kubernetes, [Specify a memory request and a memory limit](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#specify-a-memory-request-and-a-memory-limit) for examples of memory requests, and [Provisioner Configuration]({{}}) for a list of supported resources. +See [Managing Resources for Containers](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/) for details on resource types supported by Kubernetes, [Specify a memory request and a memory limit](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/#specify-a-memory-request-and-a-memory-limit) for examples of memory requests, and [Provisioner Configuration]({{}}) for a list of supported resources. ### Accelerators/GPU Resources diff --git a/website/content/en/preview/faq.md b/website/content/en/preview/faq.md index 3500660a7c33..44df132dba6d 100644 --- a/website/content/en/preview/faq.md +++ b/website/content/en/preview/faq.md @@ -21,7 +21,7 @@ Start with Karpenter's GitHub [cloudprovider](https://github.com/aws/karpenter-c By default, Karpenter uses Amazon Linux 2 images. ### Can I provide my own custom operating system images? -Karpenter has multiple mechanisms for configuring the [operating system]({{< ref "./concepts/node-templates/#spec-amiselector" >}}) for your nodes. +Karpenter has multiple mechanisms for configuring the [operating system]({{< ref "./concepts/nodeclasses/#spec-amiselector" >}}) for your nodes. ### Can Karpenter deal with workloads for mixed architecture cluster (arm vs. amd)? Karpenter is flexible to multi architecture configurations using [well known labels]({{< ref "./concepts/scheduling/#supported-labels">}}). @@ -57,11 +57,11 @@ See [Scheduling]({{< ref "./concepts/scheduling" >}}) for details on how Karpent ## Provisioning ### What features does the Karpenter provisioner support? -See [Provisioner API]({{< ref "./concepts/provisioners" >}}) for provisioner examples and descriptions of features. +See [Provisioner API]({{< ref "./concepts/nodepools" >}}) for provisioner examples and descriptions of features. ### Can I create multiple (team-based) provisioners on a cluster? Yes, provisioners can identify multiple teams based on labels. -See [Provisioner API]({{< ref "./concepts/provisioners" >}}) for details. +See [Provisioner API]({{< ref "./concepts/nodepools" >}}) for details. ### If multiple provisioners are defined, which will my pod use? @@ -88,14 +88,14 @@ However, you can use Session Manager (SSM) or EC2 Instance Connect to gain shell See [Node NotReady]({{< ref "./troubleshooting/#node-notready" >}}) troubleshooting for an example of starting an SSM session from the command line or [EC2 Instance Connect](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html) documentation to connect to nodes using SSH. Though not recommended, if you need to access Karpenter-managed nodes without AWS credentials, you can add SSH keys using AWSNodeTemplate. -See [Custom User Data]({{< ref "./concepts/node-templates/#spec-userdata" >}}) for details. +See [Custom User Data]({{< ref "./concepts/nodeclasses/#spec-userdata" >}}) for details. ### Can I set total limits of CPU and memory for a provisioner? Yes, the setting is provider-specific. See examples in [Accelerators, GPU]({{< ref "./concepts/scheduling/#accelerators-gpu-resources" >}}) Karpenter documentation. ### Can I mix spot and on-demand EC2 run types? -Yes, see [Provisioning]({{< ref "./concepts/provisioners#examples" >}}) for an example. +Yes, see [Provisioning]({{< ref "./concepts/nodepools#examples" >}}) for an example. ### Can I restrict EC2 instance types? @@ -194,7 +194,7 @@ See [Application developer]({{< ref "./concepts/#application-developer" >}}) for Yes. See [Persistent Volume Topology]({{< ref "./concepts/scheduling#persistent-volume-topology" >}}) for details. ### Can I set `--max-pods` on my nodes? -Yes, see the [KubeletConfiguration Section in the Provisioners Documentation]({{}}) to learn more. +Yes, see the [KubeletConfiguration Section in the Provisioners Documentation]({{}}) to learn more. ### Why do the Windows2019 and Windows2022 AMI families only support Windows Server Core? The difference between the Core and Full variants is that Core is a minimal OS with less components and no graphic user interface (GUI) or desktop experience. @@ -208,7 +208,7 @@ amiSelector: ## Deprovisioning ### How does Karpenter deprovision nodes? -See [Deprovisioning nodes]({{< ref "./concepts/deprovisioning" >}}) for information on how Karpenter deprovisions nodes. +See [Deprovisioning nodes]({{< ref "./concepts/disruption" >}}) for information on how Karpenter deprovisions nodes. ## Upgrading Karpenter @@ -235,9 +235,9 @@ The `startupTaints` parameter was added in v0.10.0. Helm upgrades do not upgrad ### How do I upgrade an EKS Cluster with Karpenter? -When upgrading an Amazon EKS cluster, [Karpenter's Drift feature]({{}}) can automatically upgrade the Karpenter-provisioned nodes to stay in-sync with the EKS control plane. Karpenter Drift currently needs to be enabled using a [feature gate]({{}}). Karpenter's default [AWSNodeTemplate `amiFamily` configuration]({{}}) uses the latest EKS Optimized AL2 AMI for the same major and minor version as the EKS cluster's control plane. Karpenter's AWSNodeTemplate can be configured to not use the EKS optimized AL2 AMI in favor of a custom AMI by configuring the [`amiSelector`]({{}}). If using a custom AMI, you will need to trigger the rollout of this new worker node image through the publication of a new AMI with tags matching the [`amiSelector`]({{}}), or a change to the [`amiSelector`]({{}}) field. +When upgrading an Amazon EKS cluster, [Karpenter's Drift feature]({{}}) can automatically upgrade the Karpenter-provisioned nodes to stay in-sync with the EKS control plane. Karpenter Drift currently needs to be enabled using a [feature gate]({{}}). Karpenter's default [AWSNodeTemplate `amiFamily` configuration]({{}}) uses the latest EKS Optimized AL2 AMI for the same major and minor version as the EKS cluster's control plane. Karpenter's AWSNodeTemplate can be configured to not use the EKS optimized AL2 AMI in favor of a custom AMI by configuring the [`amiSelector`]({{}}). If using a custom AMI, you will need to trigger the rollout of this new worker node image through the publication of a new AMI with tags matching the [`amiSelector`]({{}}), or a change to the [`amiSelector`]({{}}) field. -Start by [upgrading the EKS Cluster control plane](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html). After the EKS Cluster upgrade completes, Karpenter's Drift feature will detect that the Karpenter-provisioned nodes are using EKS Optimized AMIs for the previous cluster version, and [automatically cordon, drain, and replace those nodes]({{}}). To support pods moving to new nodes, follow Kubernetes best practices by setting appropriate pod [Resource Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/), and using [Pod Disruption Budgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) (PDB). Karpenter's Drift feature will spin up replacement nodes based on the pod resource requests, and will respect the PDBs when deprovisioning nodes. +Start by [upgrading the EKS Cluster control plane](https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html). After the EKS Cluster upgrade completes, Karpenter's Drift feature will detect that the Karpenter-provisioned nodes are using EKS Optimized AMIs for the previous cluster version, and [automatically cordon, drain, and replace those nodes]({{}}). To support pods moving to new nodes, follow Kubernetes best practices by setting appropriate pod [Resource Quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/), and using [Pod Disruption Budgets](https://kubernetes.io/docs/concepts/workloads/pods/disruptions/) (PDB). Karpenter's Drift feature will spin up replacement nodes based on the pod resource requests, and will respect the PDBs when deprovisioning nodes. ## Interruption Handling @@ -252,7 +252,7 @@ Karpenter's native interruption handling offers two main benefits over the stand ### Why am I receiving QueueNotFound errors when I set `aws.interruptionQueueName`? Karpenter requires a queue to exist that receives event messages from EC2 and health services in order to handle interruption messages properly for nodes. -Details on the types of events that Karpenter handles can be found in the [Interruption Handling Docs]({{< ref "./concepts/deprovisioning/#interruption" >}}). +Details on the types of events that Karpenter handles can be found in the [Interruption Handling Docs]({{< ref "./concepts/disruption/#interruption" >}}). Details on provisioning the SQS queue and EventBridge rules can be found in the [Getting Started Guide]({{< ref "./getting-started/getting-started-with-karpenter/#create-the-karpenter-infrastructure-and-iam-roles" >}}). diff --git a/website/content/en/preview/getting-started/getting-started-with-karpenter/_index.md b/website/content/en/preview/getting-started/getting-started-with-karpenter/_index.md index eb29e0b7de40..efdf7d929b38 100644 --- a/website/content/en/preview/getting-started/getting-started-with-karpenter/_index.md +++ b/website/content/en/preview/getting-started/getting-started-with-karpenter/_index.md @@ -110,9 +110,9 @@ This provisioner uses `securityGroupSelector` and `subnetSelector` to discover r We applied the tag `karpenter.sh/discovery` in the `eksctl` command above. Depending how these resources are shared between clusters, you may need to use different tagging schemes. -The `consolidation` value configures Karpenter to reduce cluster cost by removing and replacing nodes. As a result, consolidation will terminate any empty nodes on the cluster. This behavior can be disabled by leaving the value undefined or setting `consolidation.enabled` to `false`. Review the [provisioner CRD]({{}}) for more information. +The `consolidation` value configures Karpenter to reduce cluster cost by removing and replacing nodes. As a result, consolidation will terminate any empty nodes on the cluster. This behavior can be disabled by leaving the value undefined or setting `consolidation.enabled` to `false`. Review the [provisioner CRD]({{}}) for more information. -Review the [provisioner CRD]({{}}) for more information. For example, +Review the [provisioner CRD]({{}}) for more information. For example, `ttlSecondsUntilExpired` configures Karpenter to terminate nodes when a maximum age is reached. Note: This provisioner will create capacity as long as the sum of all created capacity is less than the specified limit. diff --git a/website/content/en/preview/troubleshooting.md b/website/content/en/preview/troubleshooting.md index d75b3d527d77..4498a4f15974 100644 --- a/website/content/en/preview/troubleshooting.md +++ b/website/content/en/preview/troubleshooting.md @@ -202,7 +202,7 @@ See the Karpenter [Best Practices Guide](https://aws.github.io/aws-eks-best-prac ### Missing subnetSelector and securityGroupSelector tags causes provisioning failures -Starting with Karpenter v0.5.5, if you are using Karpenter-generated launch template, provisioners require that [subnetSelector]({{}}) and [securityGroupSelector]({{}}) tags be set to match your cluster. +Starting with Karpenter v0.5.5, if you are using Karpenter-generated launch template, provisioners require that [subnetSelector]({{}}) and [securityGroupSelector]({{}}) tags be set to match your cluster. The [Provisioner]({{}}) section in the Karpenter Getting Started Guide uses the following example: ```text @@ -289,7 +289,7 @@ time=2023-06-12T19:18:15Z type=Warning reason=FailedCreatePodSandBox from=kubele By default, the number of pods on a node is limited by both the number of networking interfaces (ENIs) that may be attached to an instance type and the number of IP addresses that can be assigned to each ENI. See [IP addresses per network interface per instance type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) for a more detailed information on these instance types' limits. -If the max-pods (configured through your Provisioner [`kubeletConfiguration`]({{}})) is greater than the number of supported IPs for a given instance type, the CNI will fail to assign an IP to the pod and your pod will be left in a `ContainerCreating` state. +If the max-pods (configured through your Provisioner [`kubeletConfiguration`]({{}})) is greater than the number of supported IPs for a given instance type, the CNI will fail to assign an IP to the pod and your pod will be left in a `ContainerCreating` state. ##### Solutions @@ -297,13 +297,13 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your Provisioner -3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. For more information on pod density, view the [Pod Density Conceptual Documentation]({{}}). #### IP exhaustion in a subnet -When a node is launched by Karpenter, it is assigned to a subnet within your VPC based on the [`subnetSelector`]({{}}) value in your [`AWSNodeTemplate`]({{}})). When a subnet becomes IP address constrained, EC2 may think that it can successfully launch an instance in the subnet; however, when the CNI tries to assign IPs to the pods, there are none remaining. In this case, your pod will stay in a `ContainerCreating` state until an IP address is freed in the subnet and the CNI can assign one to the pod. +When a node is launched by Karpenter, it is assigned to a subnet within your VPC based on the [`subnetSelector`]({{}}) value in your [`AWSNodeTemplate`]({{}})). When a subnet becomes IP address constrained, EC2 may think that it can successfully launch an instance in the subnet; however, when the CNI tries to assign IPs to the pods, there are none remaining. In this case, your pod will stay in a `ContainerCreating` state until an IP address is freed in the subnet and the CNI can assign one to the pod. ##### Solutions @@ -550,7 +550,7 @@ This means that your CNI plugin is out of date. You can find instructions on how ### Node terminates before ready on failed encrypted EBS volume If you are using a custom launch template and an encrypted EBS volume, the IAM principal launching the node may not have sufficient permissions to use the KMS customer managed key (CMK) for the EC2 EBS root volume. -This issue also applies to [Block Device Mappings]({{}}) specified in the Provisioner. +This issue also applies to [Block Device Mappings]({{}}) specified in the Provisioner. In either case, this results in the node terminating almost immediately upon creation. Keep in mind that it is possible that EBS Encryption can be enabled without your knowledge. diff --git a/website/content/en/preview/upgrade-guide.md b/website/content/en/preview/upgrade-guide.md index 8f4ef1fcd4af..a606ed69a61b 100644 --- a/website/content/en/preview/upgrade-guide.md +++ b/website/content/en/preview/upgrade-guide.md @@ -131,7 +131,7 @@ Snapshot releases are tagged with the git commit hash prefixed by the Karpenter ### Upgrading to v0.30.0+ -* Karpenter will now [statically drift]({{}}) on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the `karpenter.sh/provisioner-hash` annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the `karpenter.k8s.aws/nodetemplate-hash` annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added. +* Karpenter will now [statically drift]({{}}) on both Provisioner and AWSNodeTemplate Fields. For Provisioner Static Drift, the `karpenter.sh/provisioner-hash` annotation must be present on both the Provisioner and Machine. For AWSNodeTemplate drift, the `karpenter.k8s.aws/nodetemplate-hash` annotation must be present on the AWSNodeTemplate and Machine. Karpenter will not add these annotations to pre-existing nodes, so each of these nodes will need to be recycled one time for the annotations to be added. * Karpenter will now fail validation on AWSNodeTemplates and Provisioner `spec.provider` that have `amiSelectors`, `subnetSelectors`, or `securityGroupSelectors` set with a combination of id selectors (`aws-ids`, `aws::ids`) and other selectors. * Karpenter now statically sets the `securityContext` at both the pod and container-levels and doesn't allow override values to be passed through the helm chart. This change was made to adhere to [Restricted Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted), which follows pod hardening best practices. @@ -215,7 +215,7 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh * The `karpenter_allocation_controller_scheduling_duration_seconds` metric name changed to `karpenter_provisioner_scheduling_duration_seconds` ### Upgrading to v0.26.0+ -* The `karpenter.sh/do-not-evict` annotation no longer blocks node termination when running `kubectl delete node`. This annotation on pods will only block automatic deprovisioning that is considered "voluntary," that is, disruptions that can be avoided. Disruptions that Karpenter deems as "involuntary" and will ignore the `karpenter.sh/do-not-evict` annotation include spot interruption and manual deletion of the node. See [Disabling Deprovisioning]({{}}) for more details. +* The `karpenter.sh/do-not-evict` annotation no longer blocks node termination when running `kubectl delete node`. This annotation on pods will only block automatic deprovisioning that is considered "voluntary," that is, disruptions that can be avoided. Disruptions that Karpenter deems as "involuntary" and will ignore the `karpenter.sh/do-not-evict` annotation include spot interruption and manual deletion of the node. See [Disabling Deprovisioning]({{}}) for more details. * Default resources `requests` and `limits` are removed from the Karpenter's controller deployment through the Helm chart. If you have not set custom resource `requests` or `limits` in your helm values and are using Karpenter's defaults, you will now need to set these values in your helm chart deployment. * The `controller.image` value in the helm chart has been broken out to a map consisting of `controller.image.repository`, `controller.image.tag`, and `controller.image.digest`. If manually overriding the `controller.image`, you will need to update your values to the new design. @@ -235,7 +235,7 @@ kubectl delete mutatingwebhookconfigurations defaulting.webhook.karpenter.sh ### Upgrading to v0.19.0+ * The karpenter webhook and controller containers are combined into a single binary, which requires changes to the helm chart. If your Karpenter installation (helm or otherwise) currently customizes the karpenter webhook, your deployment tooling may require minor changes. -* Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling `aws.interruptionQueueName`. For more details on Karpenter's interruption handling, see the [Interruption Handling Docs]({{< ref "./concepts/deprovisioning/#interruption" >}}). For common questions on the migration process, see the [FAQ]({{< ref "./faq/#interruption-handling" >}}) +* Karpenter now supports native interruption handling. If you were previously using Node Termination Handler for spot interruption handling and health events, you will need to remove the component from your cluster before enabling `aws.interruptionQueueName`. For more details on Karpenter's interruption handling, see the [Interruption Handling Docs]({{< ref "./concepts/disruption/#interruption" >}}). For common questions on the migration process, see the [FAQ]({{< ref "./faq/#interruption-handling" >}}) * Instance category defaults are now explicitly persisted in the Provisioner, rather than handled implicitly in memory. By default, Provisioners will limit instance category to c,m,r. If any instance type constraints are applied, it will override this default. If you have created Provisioners in the past with unconstrained instance type, family, or category, Karpenter will now more flexibly use instance types than before. If you would like to apply these constraints, they must be included in the Provisioner CRD. * Karpenter CRD raw YAML URLs have migrated from `https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}charts/karpenter/crds/...` to `https://raw.githubusercontent.com/aws/karpenter{{< githubRelRef >}}pkg/apis/crds/...`. If you reference static Karpenter CRDs or rely on `kubectl replace -f` to apply these CRDs from their remote location, you will need to migrate to the new location. * Pods without an ownerRef (also called "controllerless" or "naked" pods) will now be evicted by default during node termination and consolidation. Users can prevent controllerless pods from being voluntarily disrupted by applying the `karpenter.sh/do-not-evict: "true"` annotation to the pods in question. @@ -305,7 +305,7 @@ aws ec2 delete-launch-template --launch-template-id operator: Exists ``` -* v0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the `ec2:DescribeImages` permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the [Custom AMI documentation here]({{}}) to get started +* v0.14.0 introduces support for custom AMIs without the need for an entire launch template. You must add the `ec2:DescribeImages` permission to the Karpenter Controller Role for this feature to work. This permission is needed for Karpenter to discover custom images specified. Read the [Custom AMI documentation here]({{}}) to get started * v0.14.0 adds an an additional default toleration (CriticalAddonOnly=Exists) to the Karpenter helm chart. This may cause Karpenter to run on nodes with that use this Taint which previously would not have been schedulable. This can be overridden by using `--set tolerations[0]=null`. * v0.14.0 deprecates the `AWS_ENI_LIMITED_POD_DENSITY` environment variable in-favor of specifying `spec.kubeletConfiguration.maxPods` on the Provisioner. `AWS_ENI_LIMITED_POD_DENSITY` will continue to work when `maxPods` is not set on the Provisioner. If `maxPods` is set, it will override `AWS_ENI_LIMITED_POD_DENSITY` on that specific Provisioner. diff --git a/website/static/_redirects b/website/static/_redirects index ea5f4e803645..239a5d444163 100644 --- a/website/static/_redirects +++ b/website/static/_redirects @@ -5,3 +5,6 @@ /v0.26/getting-started/getting-started-with-karpenter/* /v0.26/getting-started/getting-started-with-eksctl/:splat /v0.27/getting-started/getting-started-with-eksctl/* /v0.27/getting-started/getting-started-with-karpenter/:splat /v0.28/getting-started/getting-started-with-eksctl/* /v0.28/getting-started/getting-started-with-karpenter/:splat +/preview/concepts/provisioners /preview/concepts/nodepools +/preview/concepts/node-templates /preview/concepts/nodeclasses +/preview/concepts/deprovisioning /preview/concepts/disruption