diff --git a/website/content/en/preview/getting-started/migrating-from-cas/_index.md b/website/content/en/preview/getting-started/migrating-from-cas/_index.md index 4dc8fe8a6a66..96a20267bbb3 100644 --- a/website/content/en/preview/getting-started/migrating-from-cas/_index.md +++ b/website/content/en/preview/getting-started/migrating-from-cas/_index.md @@ -16,21 +16,26 @@ We will make the following assumptions in this guide * Your workloads have pod disruption budgets that adhere to [EKS best practices](https://aws.github.io/aws-eks-best-practices/karpenter/) * Your cluster has an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for service accounts -This guide will also assume you have the `aws` CLI installed. +This guide will also assume you have the `aws` CLI and Helm client installed. You can also perform many of these steps in the console, but we will use the command line for simplicity. ## Set environment variables +Set the Karpenter and Kubernetes version. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. ```bash -KARPENTER_NAMESPACE=kube-system +KARPENTER_NAMESPACE="kube-system" +KARPENTER_VERSION="{{< param "latest_release_version" >}}" +K8S_VERSION="{{< param "latest_k8s_version" >}}" CLUSTER_NAME= -K8S_VERSION=1.28 ``` Set other variables from your cluster configuration. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} +{{% alert title="Warning" color="warning" %}} +If you open a new shell to run steps in this procedure, you need to set the environment variables again. +{{% /alert %}} ## Create IAM roles @@ -40,7 +45,7 @@ Use CloudFormation to set up the infrastructure needed by the existing EKS clust - **Instance Profiles**: Attaches necessary permissions to EC2 instances, allowing them to join the cluster and participate in automated scaling as managed by Karpenter. - **Interruption Queue and Policies**: Setup Amazon SQS queue and Event Rules for handling interruption notifications from AWS services related to EC2 Spot instances and AWS Health events. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which requires an OIDC endpoint. @@ -48,26 +53,48 @@ The controller will be using [IAM Roles for Service Accounts (IRSA)](https://doc If you have another option for using IAM credentials with workloads (e.g. [Amazon EKS Pod Identity Agent](https://github.com/aws/eks-pod-identity-agent)) your steps will be different. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} ## Add tags to subnets and security groups -We need to add tags to our nodegroup subnets so Karpenter will know which subnets to use. +In order for Karpenter to know which [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to use, we need to add appropriate tags to the nodegroup subnets and security groups. + +### Tag nodegroup subnets + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} + +This loop ensures that Karpenter will be aware of which subnets are associated with each nodegroup by tagging them with karpenter.sh/discovery. + +### Tag security groups + +If your EKS setup is configured to use cluster security group and additional security groups, execute the following commands to tag them for Karpenter discovery: + +```bash +SECURITY_GROUPS=$(aws eks describe-cluster \ + --name "${CLUSTER_NAME}" \ + --query "cluster.resourcesVpcConfig" \ + --output json | jq -r '[.clusterSecurityGroupId] + .securityGroupIds | join(" ")') + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources ${SECURITY_GROUPS} +``` + +If your setup uses security groups from the Launch template of a managed nodegroup, execute the following: -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} +Note that this command will only tag the security groups for the first nodegroup in the cluster. If you have multiple nodegroups groups, you will need to decide which ones Karpenter should use. -Add tags to our security groups. -This command only tags the security groups for the first nodegroup in the cluster. -If you have multiple nodegroups or multiple security groups you will need to decide which one Karpenter should use. +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} + +Alternatively, the subnets and security groups can also be defined in the [NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/) definition by specifying the [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnets) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to be used. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} ## Update aws-auth ConfigMap We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the `aws-auth` ConfigMap in the cluster. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} You will need to add a section to the mapRoles that looks something like this. Replace the `${AWS_PARTITION}` variable with the account partition, `${AWS_ACCOUNT_ID}` variable with your account ID, and `${CLUSTER_NAME}` variable with the cluster name, but do not replace the `{{EC2PrivateDNSName}}`. @@ -87,12 +114,7 @@ The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group. ## Deploy Karpenter - -First, set the Karpenter release you want to deploy. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. - -```bash -export KARPENTER_VERSION="{{< param "latest_release_version" >}}" -``` +To deploy Karpenter, you can use Helm, which simplifies the installation process by handling Karpenter’s dependencies and configuration files automatically. The Helm command provided below will also incorporate any customized settings, such as node affinity, to align with your specific deployment needs. ### Set Node Affinity for Karpenter @@ -120,13 +142,26 @@ EOF Now that you have prepared the node affinity configuration, you can proceed to install Karpenter using Helm. This command includes the affinity settings along with other necessary configurations: -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} + +Expected output: +```bash +Release "karpenter" does not exist. Installing it now. +Pulled: public.ecr.aws/karpenter/karpenter:1.0.5 +Digest: sha256:98382d6406a3c85711269112fbb337c056d4debabaefb936db2d10137b58bd1b +NAME: karpenter +LAST DEPLOYED: Wed Nov 6 16:51:41 2024 +NAMESPACE: kube-system +STATUS: deployed +REVISION: 1 +TEST SUITE: None +``` ## Create default NodePool We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree{{< githubRelRef >}}examples/v1) for specific needs. -{{% script file="./content/en/preview/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} ## Set nodeAffinity for critical workloads (optional) @@ -175,15 +210,16 @@ Or, if you have multiple single-AZ node groups, we suggest a minimum of 1 instan If you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured. {{% /alert %}} + ## Verify Karpenter As nodegroup nodes are drained you can verify that Karpenter is creating nodes for your workloads. ```bash -kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenter +kubectl logs -f -n $KARPENTER_NAMESPACE -c controller -l app.kubernetes.io/name=karpenter ``` -You should also see new nodes created in your cluster as the old nodes are removed +You should also see new nodes created in your cluster as the old nodes are removed. ```bash kubectl get nodes diff --git a/website/content/en/preview/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh b/website/content/en/preview/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh index c63bde3b78dc..764b3917e3f4 100644 --- a/website/content/en/preview/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh +++ b/website/content/en/preview/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh @@ -5,13 +5,6 @@ LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ --nodegroup-name "${NODEGROUP}" --query 'nodegroup.launchTemplate.{id:id,version:version}' \ --output text | tr -s "\t" ",") -# If your EKS setup is configured to use only Cluster security group, then please execute - - -SECURITY_GROUPS=$(aws eks describe-cluster \ - --name "${CLUSTER_NAME}" --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text) - -# If your setup uses the security groups in the Launch template of a managed node group, then : - SECURITY_GROUPS="$(aws ec2 describe-launch-template-versions \ --launch-template-id "${LAUNCH_TEMPLATE%,*}" --versions "${LAUNCH_TEMPLATE#*,}" \ --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ diff --git a/website/content/en/preview/getting-started/migrating-from-cas/scripts/step07-deploy.sh b/website/content/en/preview/getting-started/migrating-from-cas/scripts/step07-deploy.sh index be8955e890e9..01128a09c884 100644 --- a/website/content/en/preview/getting-started/migrating-from-cas/scripts/step07-deploy.sh +++ b/website/content/en/preview/getting-started/migrating-from-cas/scripts/step07-deploy.sh @@ -1,6 +1,7 @@ helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set "settings.clusterName=${CLUSTER_NAME}" \ --set "settings.interruptionQueue=${CLUSTER_NAME}" \ + --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ --set controller.resources.requests.cpu=1 \ --set controller.resources.requests.memory=1Gi \ --set controller.resources.limits.cpu=1 \ diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/_index.md b/website/content/en/v0.32/getting-started/migrating-from-cas/_index.md index 83134c7c83a9..96a20267bbb3 100644 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/_index.md +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/_index.md @@ -10,19 +10,22 @@ This guide will show you how to switch from the [Kubernetes Cluster Autoscaler]( We will make the following assumptions in this guide * You will use an existing EKS cluster -* You will use existing VPC and subnets -* You will use existing security groups +* You will use an existing VPC and subnets +* You will use an existing security groups * Your nodes are part of one or more node groups * Your workloads have pod disruption budgets that adhere to [EKS best practices](https://aws.github.io/aws-eks-best-practices/karpenter/) * Your cluster has an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for service accounts -This guide will also assume you have the `aws` CLI installed. +This guide will also assume you have the `aws` CLI and Helm client installed. You can also perform many of these steps in the console, but we will use the command line for simplicity. -Set a variable for your cluster name. +## Set environment variables +Set the Karpenter and Kubernetes version. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. ```bash -KARPENTER_NAMESPACE=karpenter +KARPENTER_NAMESPACE="kube-system" +KARPENTER_VERSION="{{< param "latest_release_version" >}}" +K8S_VERSION="{{< param "latest_k8s_version" >}}" CLUSTER_NAME= ``` @@ -30,45 +33,68 @@ Set other variables from your cluster configuration. {{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} -Use that information to create our IAM roles, inline policy, and trust relationship. +{{% alert title="Warning" color="warning" %}} +If you open a new shell to run steps in this procedure, you need to set the environment variables again. +{{% /alert %}} ## Create IAM roles -To get started with our migration we first need to create two new IAM roles for nodes provisioned with Karpenter and the Karpenter controller. - -To create the Karpenter node role we will use the following policy and commands. - -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-node-iam.sh" language="bash" %}} +Use CloudFormation to set up the infrastructure needed by the existing EKS cluster. See [CloudFormation]({{< relref "../../reference/cloudformation/" >}}) for a complete description of what `cloudformation.yaml` does for Karpenter. The provided `cloudformation.yaml` template simplifies this setup by creating and configuring all necessary resources, including: -Now attach the required policies to the role + - **IAM Roles and Policies**: Grants Karpenter permissions to interact with EKS, autoscaling, and EC2 services, enabling it to manage nodes dynamically. + - **Instance Profiles**: Attaches necessary permissions to EC2 instances, allowing them to join the cluster and participate in automated scaling as managed by Karpenter. + - **Interruption Queue and Policies**: Setup Amazon SQS queue and Event Rules for handling interruption notifications from AWS services related to EC2 Spot instances and AWS Health events. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-node-policies.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which requires an OIDC endpoint. -If you have another option for using IAM credentials with workloads (e.g. [kube2iam](https://github.com/jtblin/kube2iam)) your steps will be different. +If you have another option for using IAM credentials with workloads (e.g. [Amazon EKS Pod Identity Agent](https://github.com/aws/eks-pod-identity-agent)) your steps will be different. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh" language="bash" %}} + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} ## Add tags to subnets and security groups -We need to add tags to our nodegroup subnets so Karpenter will know which subnets to use. +In order for Karpenter to know which [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to use, we need to add appropriate tags to the nodegroup subnets and security groups. + +### Tag nodegroup subnets -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} -Add tags to our security groups. -This command only tags the security groups for the first nodegroup in the cluster. -If you have multiple nodegroups or multiple security groups you will need to decide which one Karpenter should use. +This loop ensures that Karpenter will be aware of which subnets are associated with each nodegroup by tagging them with karpenter.sh/discovery. + +### Tag security groups + +If your EKS setup is configured to use cluster security group and additional security groups, execute the following commands to tag them for Karpenter discovery: + +```bash +SECURITY_GROUPS=$(aws eks describe-cluster \ + --name "${CLUSTER_NAME}" \ + --query "cluster.resourcesVpcConfig" \ + --output json | jq -r '[.clusterSecurityGroupId] + .securityGroupIds | join(" ")') + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources ${SECURITY_GROUPS} +``` + +If your setup uses security groups from the Launch template of a managed nodegroup, execute the following: + +Note that this command will only tag the security groups for the first nodegroup in the cluster. If you have multiple nodegroups groups, you will need to decide which ones Karpenter should use. + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} + +Alternatively, the subnets and security groups can also be defined in the [NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/) definition by specifying the [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnets) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to be used. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh" language="bash" %}} ## Update aws-auth ConfigMap We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the `aws-auth` ConfigMap in the cluster. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} You will need to add a section to the mapRoles that looks something like this. Replace the `${AWS_PARTITION}` variable with the account partition, `${AWS_ACCOUNT_ID}` variable with your account ID, and `${CLUSTER_NAME}` variable with the cluster name, but do not replace the `{{EC2PrivateDNSName}}`. @@ -88,53 +114,54 @@ The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group. ## Deploy Karpenter +To deploy Karpenter, you can use Helm, which simplifies the installation process by handling Karpenter’s dependencies and configuration files automatically. The Helm command provided below will also incorporate any customized settings, such as node affinity, to align with your specific deployment needs. -First set the Karpenter release you want to deploy. - -```bash -export KARPENTER_VERSION=v0.32.10 -``` +### Set Node Affinity for Karpenter -We can now generate a full Karpenter deployment yaml from the helm chart. +To optimize resource usage and ensure that Karpenter schedules its pods on nodes within a specific, existing node group, it is essential to configure node affinity. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh" language="bash" %}} +Create a file named karpenter-node-affinity.yaml to define the node affinity settings and specify the node group where you want Karpenter to deploy. -Modify the following lines in the karpenter.yaml file. +Be sure to replace `${NODEGROUP}` with the actual name of your node group. -### Set node affinity - -Edit the karpenter.yaml file and find the karpenter deployment affinity rules. -Modify the affinity so karpenter will run on one of the existing node group nodes. - -The rules should look something like this. -Modify the value to match your `$NODEGROUP`, one node group per line. - -```yaml +```bash +cat < karpenter-node-affinity.yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - - matchExpressions: - - key: karpenter.sh/nodepool - operator: DoesNotExist - - key: eks.amazonaws.com/nodegroup - operator: In - values: - - ${NODEGROUP} - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - topologyKey: "kubernetes.io/hostname" + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist + - key: eks.amazonaws.com/nodegroup + operator: In + values: + - "${NODEGROUP}" +EOF ``` -Now that our deployment is ready we can create the karpenter namespace, create the NodePool CRD, and then deploy the rest of the karpenter resources. +Now that you have prepared the node affinity configuration, you can proceed to install Karpenter using Helm. This command includes the affinity settings along with other necessary configurations: -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-deploy.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} + +Expected output: +```bash +Release "karpenter" does not exist. Installing it now. +Pulled: public.ecr.aws/karpenter/karpenter:1.0.5 +Digest: sha256:98382d6406a3c85711269112fbb337c056d4debabaefb936db2d10137b58bd1b +NAME: karpenter +LAST DEPLOYED: Wed Nov 6 16:51:41 2024 +NAMESPACE: kube-system +STATUS: deployed +REVISION: 1 +TEST SUITE: None +``` ## Create default NodePool -We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree/v0.32.10/examples/v1beta1) for specific needs. +We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree{{< githubRelRef >}}examples/v1) for specific needs. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} ## Set nodeAffinity for critical workloads (optional) @@ -165,7 +192,7 @@ affinity: Now that karpenter is running we can disable the cluster autoscaler. To do that we will scale the number of replicas to zero. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh" language="bash" %}} To get rid of the instances that were added from the node group we can scale our nodegroup down to a minimum size to support Karpenter and other critical services. @@ -173,25 +200,26 @@ To get rid of the instances that were added from the node group we can scale our If you have a single multi-AZ node group, we suggest a minimum of 2 instances. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh" language="bash" %}} Or, if you have multiple single-AZ node groups, we suggest a minimum of 1 instance each. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh" language="bash" %}} {{% alert title="Note" color="warning" %}} If you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured. {{% /alert %}} + ## Verify Karpenter As nodegroup nodes are drained you can verify that Karpenter is creating nodes for your workloads. ```bash -kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenter +kubectl logs -f -n $KARPENTER_NAMESPACE -c controller -l app.kubernetes.io/name=karpenter ``` -You should also see new nodes created in your cluster as the old nodes are removed +You should also see new nodes created in your cluster as the old nodes are removed. ```bash kubectl get nodes diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step01-env.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step01-env.sh index 6276d7aa1439..3ee74647cb3c 100644 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step01-env.sh +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step01-env.sh @@ -1,10 +1,10 @@ AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov AWS_REGION="$(aws configure list | grep region | tr -s " " | cut -d" " -f3)" -OIDC_ENDPOINT="$(aws eks describe-cluster --name ${CLUSTER_NAME} \ +OIDC_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" \ --query "cluster.identity.oidc.issuer" --output text)" AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \ --output text) -K8S_VERSION=1.28 +TEMPOUT="$(mktemp)" ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)" AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)" GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)" diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh new file mode 100644 index 000000000000..dcf1cb658dd4 --- /dev/null +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh @@ -0,0 +1,6 @@ +curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \ +&& aws cloudformation deploy \ + --stack-name "Karpenter-${CLUSTER_NAME}" \ + --template-file "${TEMPOUT}" \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameter-overrides "ClusterName=${CLUSTER_NAME}" diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-node-iam.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-node-iam.sh deleted file mode 100644 index 07cd4defce11..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step02-node-iam.sh +++ /dev/null @@ -1,15 +0,0 @@ -echo '{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] -}' > node-trust-policy.json - -aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://node-trust-policy.json diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh new file mode 100644 index 000000000000..592e4575d3db --- /dev/null +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh @@ -0,0 +1,26 @@ +cat << EOF > controller-trust-policy.json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", + "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" + } + } + } + ] +} +EOF + +aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --assume-role-policy-document file://controller-trust-policy.json + +aws iam attach-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-node-policies.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-node-policies.sh deleted file mode 100644 index d57f79039d04..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step03-node-policies.sh +++ /dev/null @@ -1,11 +0,0 @@ -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh deleted file mode 100644 index 70c59e826393..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh +++ /dev/null @@ -1,143 +0,0 @@ -cat << EOF > controller-trust-policy.json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", - "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" - } - } - } - ] -} -EOF - -aws iam create-role --role-name KarpenterControllerRole-${CLUSTER_NAME} \ - --assume-role-policy-document file://controller-trust-policy.json - -cat << EOF > controller-policy.json -{ - "Statement": [ - { - "Action": [ - "ssm:GetParameter", - "ec2:DescribeImages", - "ec2:RunInstances", - "ec2:DescribeSubnets", - "ec2:DescribeSecurityGroups", - "ec2:DescribeLaunchTemplates", - "ec2:DescribeInstances", - "ec2:DescribeInstanceTypes", - "ec2:DescribeInstanceTypeOfferings", - "ec2:DescribeAvailabilityZones", - "ec2:DeleteLaunchTemplate", - "ec2:CreateTags", - "ec2:CreateLaunchTemplate", - "ec2:CreateFleet", - "ec2:DescribeSpotPriceHistory", - "pricing:GetProducts" - ], - "Effect": "Allow", - "Resource": "*", - "Sid": "Karpenter" - }, - { - "Action": "ec2:TerminateInstances", - "Condition": { - "StringLike": { - "ec2:ResourceTag/karpenter.sh/nodepool": "*" - } - }, - "Effect": "Allow", - "Resource": "*", - "Sid": "ConditionalEC2Termination" - }, - { - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}", - "Sid": "PassNodeIAMRole" - }, - { - "Effect": "Allow", - "Action": "eks:DescribeCluster", - "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}", - "Sid": "EKSClusterEndpointLookup" - }, - { - "Sid": "AllowScopedInstanceProfileCreationActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:CreateInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileTagActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:TagInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}", - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*", - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:AddRoleToInstanceProfile", - "iam:RemoveRoleFromInstanceProfile", - "iam:DeleteInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowInstanceProfileReadActions", - "Effect": "Allow", - "Resource": "*", - "Action": "iam:GetInstanceProfile" - } - ], - "Version": "2012-10-17" -} -EOF - -aws iam put-role-policy --role-name KarpenterControllerRole-${CLUSTER_NAME} \ - --policy-name KarpenterControllerPolicy-${CLUSTER_NAME} \ - --policy-document file://controller-policy.json diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh similarity index 100% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh similarity index 68% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh index c63bde3b78dc..764b3917e3f4 100644 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh @@ -5,13 +5,6 @@ LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ --nodegroup-name "${NODEGROUP}" --query 'nodegroup.launchTemplate.{id:id,version:version}' \ --output text | tr -s "\t" ",") -# If your EKS setup is configured to use only Cluster security group, then please execute - - -SECURITY_GROUPS=$(aws eks describe-cluster \ - --name "${CLUSTER_NAME}" --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text) - -# If your setup uses the security groups in the Launch template of a managed node group, then : - SECURITY_GROUPS="$(aws ec2 describe-launch-template-versions \ --launch-template-id "${LAUNCH_TEMPLATE%,*}" --versions "${LAUNCH_TEMPLATE#*,}" \ --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh deleted file mode 100644 index de972ea2bddd..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh +++ /dev/null @@ -1,6 +0,0 @@ -for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \ - --query 'nodegroups' --output text); do aws ec2 create-tags \ - --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ - --resources $(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \ - --nodegroup-name $NODEGROUP --query 'nodegroup.subnets' --output text ) -done diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh similarity index 100% rename from website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh deleted file mode 100644 index 397e40904cee..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh +++ /dev/null @@ -1,22 +0,0 @@ -NODEGROUP=$(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \ - --query 'nodegroups[0]' --output text) - -LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \ - --nodegroup-name ${NODEGROUP} --query 'nodegroup.launchTemplate.{id:id,version:version}' \ - --output text | tr -s "\t" ",") - -# If your EKS setup is configured to use only Cluster security group, then please execute - - -SECURITY_GROUPS=$(aws eks describe-cluster \ - --name ${CLUSTER_NAME} --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text) - -# If your setup uses the security groups in the Launch template of a managed node group, then : - -SECURITY_GROUPS=$(aws ec2 describe-launch-template-versions \ - --launch-template-id ${LAUNCH_TEMPLATE%,*} --versions ${LAUNCH_TEMPLATE#*,} \ - --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ - --output text) - -aws ec2 create-tags \ - --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ - --resources ${SECURITY_GROUPS} diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step07-deploy.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step07-deploy.sh new file mode 100644 index 000000000000..01128a09c884 --- /dev/null +++ b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step07-deploy.sh @@ -0,0 +1,10 @@ +helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set "settings.clusterName=${CLUSTER_NAME}" \ + --set "settings.interruptionQueue=${CLUSTER_NAME}" \ + --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + -f karpenter-node-affinity.yaml \ + --wait diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh deleted file mode 100644 index be281d2d14f2..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh +++ /dev/null @@ -1,8 +0,0 @@ -helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" \ - --set "settings.clusterName=${CLUSTER_NAME}" \ - --set "settings.interruptionQueue=${CLUSTER_NAME}" \ - --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ - --set controller.resources.requests.cpu=1 \ - --set controller.resources.requests.memory=1Gi \ - --set controller.resources.limits.cpu=1 \ - --set controller.resources.limits.memory=1Gi > karpenter.yaml diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step09-deploy.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step09-deploy.sh deleted file mode 100644 index 057143d3645e..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step09-deploy.sh +++ /dev/null @@ -1,14 +0,0 @@ -kubectl create namespace "${KARPENTER_NAMESPACE}" || true -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_provisioners.yaml -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_awsnodetemplates.yaml -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_machines.yaml -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml -kubectl create -f \ - https://raw.githubusercontent.com/aws/karpenter-provider-aws/${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml -kubectl apply -f karpenter.yaml diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh similarity index 100% rename from website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh similarity index 100% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh similarity index 100% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh rename to website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh deleted file mode 100644 index d88a6d5c7236..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh +++ /dev/null @@ -1,5 +0,0 @@ -for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \ - --query 'nodegroups' --output text); do aws eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \ - --nodegroup-name ${NODEGROUP} \ - --scaling-config "minSize=1,maxSize=1,desiredSize=1" -done diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh b/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh deleted file mode 100644 index 51ad964c28a7..000000000000 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh +++ /dev/null @@ -1,3 +0,0 @@ -aws eks update-nodegroup-config --cluster-name ${CLUSTER_NAME} \ - --nodegroup-name ${NODEGROUP} \ - --scaling-config "minSize=2,maxSize=2,desiredSize=2" diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/_index.md b/website/content/en/v0.36/getting-started/migrating-from-cas/_index.md index 75160dab6732..96a20267bbb3 100644 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/_index.md +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/_index.md @@ -10,19 +10,22 @@ This guide will show you how to switch from the [Kubernetes Cluster Autoscaler]( We will make the following assumptions in this guide * You will use an existing EKS cluster -* You will use existing VPC and subnets -* You will use existing security groups +* You will use an existing VPC and subnets +* You will use an existing security groups * Your nodes are part of one or more node groups * Your workloads have pod disruption budgets that adhere to [EKS best practices](https://aws.github.io/aws-eks-best-practices/karpenter/) * Your cluster has an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for service accounts -This guide will also assume you have the `aws` CLI installed. +This guide will also assume you have the `aws` CLI and Helm client installed. You can also perform many of these steps in the console, but we will use the command line for simplicity. -Set a variable for your cluster name. +## Set environment variables +Set the Karpenter and Kubernetes version. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. ```bash -KARPENTER_NAMESPACE=kube-system +KARPENTER_NAMESPACE="kube-system" +KARPENTER_VERSION="{{< param "latest_release_version" >}}" +K8S_VERSION="{{< param "latest_k8s_version" >}}" CLUSTER_NAME= ``` @@ -30,45 +33,68 @@ Set other variables from your cluster configuration. {{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} -Use that information to create our IAM roles, inline policy, and trust relationship. +{{% alert title="Warning" color="warning" %}} +If you open a new shell to run steps in this procedure, you need to set the environment variables again. +{{% /alert %}} ## Create IAM roles -To get started with our migration we first need to create two new IAM roles for nodes provisioned with Karpenter and the Karpenter controller. - -To create the Karpenter node role we will use the following policy and commands. - -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-node-iam.sh" language="bash" %}} +Use CloudFormation to set up the infrastructure needed by the existing EKS cluster. See [CloudFormation]({{< relref "../../reference/cloudformation/" >}}) for a complete description of what `cloudformation.yaml` does for Karpenter. The provided `cloudformation.yaml` template simplifies this setup by creating and configuring all necessary resources, including: -Now attach the required policies to the role + - **IAM Roles and Policies**: Grants Karpenter permissions to interact with EKS, autoscaling, and EC2 services, enabling it to manage nodes dynamically. + - **Instance Profiles**: Attaches necessary permissions to EC2 instances, allowing them to join the cluster and participate in automated scaling as managed by Karpenter. + - **Interruption Queue and Policies**: Setup Amazon SQS queue and Event Rules for handling interruption notifications from AWS services related to EC2 Spot instances and AWS Health events. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-node-policies.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which requires an OIDC endpoint. -If you have another option for using IAM credentials with workloads (e.g. [kube2iam](https://github.com/jtblin/kube2iam)) your steps will be different. +If you have another option for using IAM credentials with workloads (e.g. [Amazon EKS Pod Identity Agent](https://github.com/aws/eks-pod-identity-agent)) your steps will be different. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh" language="bash" %}} + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} ## Add tags to subnets and security groups -We need to add tags to our nodegroup subnets so Karpenter will know which subnets to use. +In order for Karpenter to know which [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to use, we need to add appropriate tags to the nodegroup subnets and security groups. + +### Tag nodegroup subnets -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} -Add tags to our security groups. -This command only tags the security groups for the first nodegroup in the cluster. -If you have multiple nodegroups or multiple security groups you will need to decide which one Karpenter should use. +This loop ensures that Karpenter will be aware of which subnets are associated with each nodegroup by tagging them with karpenter.sh/discovery. + +### Tag security groups + +If your EKS setup is configured to use cluster security group and additional security groups, execute the following commands to tag them for Karpenter discovery: + +```bash +SECURITY_GROUPS=$(aws eks describe-cluster \ + --name "${CLUSTER_NAME}" \ + --query "cluster.resourcesVpcConfig" \ + --output json | jq -r '[.clusterSecurityGroupId] + .securityGroupIds | join(" ")') + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources ${SECURITY_GROUPS} +``` + +If your setup uses security groups from the Launch template of a managed nodegroup, execute the following: + +Note that this command will only tag the security groups for the first nodegroup in the cluster. If you have multiple nodegroups groups, you will need to decide which ones Karpenter should use. + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} + +Alternatively, the subnets and security groups can also be defined in the [NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/) definition by specifying the [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnets) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to be used. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh" language="bash" %}} ## Update aws-auth ConfigMap We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the `aws-auth` ConfigMap in the cluster. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} You will need to add a section to the mapRoles that looks something like this. Replace the `${AWS_PARTITION}` variable with the account partition, `${AWS_ACCOUNT_ID}` variable with your account ID, and `${CLUSTER_NAME}` variable with the cluster name, but do not replace the `{{EC2PrivateDNSName}}`. @@ -88,53 +114,54 @@ The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group. ## Deploy Karpenter +To deploy Karpenter, you can use Helm, which simplifies the installation process by handling Karpenter’s dependencies and configuration files automatically. The Helm command provided below will also incorporate any customized settings, such as node affinity, to align with your specific deployment needs. -First set the Karpenter release you want to deploy. - -```bash -export KARPENTER_VERSION="0.36.4" -``` +### Set Node Affinity for Karpenter -We can now generate a full Karpenter deployment yaml from the Helm chart. +To optimize resource usage and ensure that Karpenter schedules its pods on nodes within a specific, existing node group, it is essential to configure node affinity. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh" language="bash" %}} +Create a file named karpenter-node-affinity.yaml to define the node affinity settings and specify the node group where you want Karpenter to deploy. -Modify the following lines in the karpenter.yaml file. +Be sure to replace `${NODEGROUP}` with the actual name of your node group. -### Set node affinity - -Edit the karpenter.yaml file and find the karpenter deployment affinity rules. -Modify the affinity so karpenter will run on one of the existing node group nodes. - -The rules should look something like this. -Modify the value to match your `$NODEGROUP`, one node group per line. - -```yaml +```bash +cat < karpenter-node-affinity.yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - - matchExpressions: - - key: karpenter.sh/nodepool - operator: DoesNotExist - - key: eks.amazonaws.com/nodegroup - operator: In - values: - - ${NODEGROUP} - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - topologyKey: "kubernetes.io/hostname" + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist + - key: eks.amazonaws.com/nodegroup + operator: In + values: + - "${NODEGROUP}" +EOF ``` -Now that our deployment is ready we can create the karpenter namespace, create the NodePool CRD, and then deploy the rest of the karpenter resources. +Now that you have prepared the node affinity configuration, you can proceed to install Karpenter using Helm. This command includes the affinity settings along with other necessary configurations: -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-deploy.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} + +Expected output: +```bash +Release "karpenter" does not exist. Installing it now. +Pulled: public.ecr.aws/karpenter/karpenter:1.0.5 +Digest: sha256:98382d6406a3c85711269112fbb337c056d4debabaefb936db2d10137b58bd1b +NAME: karpenter +LAST DEPLOYED: Wed Nov 6 16:51:41 2024 +NAMESPACE: kube-system +STATUS: deployed +REVISION: 1 +TEST SUITE: None +``` ## Create default NodePool -We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree/v0.36.4/examples/v1beta1) for specific needs. +We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree{{< githubRelRef >}}examples/v1) for specific needs. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} ## Set nodeAffinity for critical workloads (optional) @@ -165,7 +192,7 @@ affinity: Now that karpenter is running we can disable the cluster autoscaler. To do that we will scale the number of replicas to zero. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh" language="bash" %}} To get rid of the instances that were added from the node group we can scale our nodegroup down to a minimum size to support Karpenter and other critical services. @@ -173,25 +200,26 @@ To get rid of the instances that were added from the node group we can scale our If you have a single multi-AZ node group, we suggest a minimum of 2 instances. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh" language="bash" %}} Or, if you have multiple single-AZ node groups, we suggest a minimum of 1 instance each. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh" language="bash" %}} {{% alert title="Note" color="warning" %}} If you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured. {{% /alert %}} + ## Verify Karpenter As nodegroup nodes are drained you can verify that Karpenter is creating nodes for your workloads. ```bash -kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenter +kubectl logs -f -n $KARPENTER_NAMESPACE -c controller -l app.kubernetes.io/name=karpenter ``` -You should also see new nodes created in your cluster as the old nodes are removed +You should also see new nodes created in your cluster as the old nodes are removed. ```bash kubectl get nodes diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step01-env.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step01-env.sh index b3ee0fa3d0ef..3ee74647cb3c 100644 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step01-env.sh +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step01-env.sh @@ -4,7 +4,7 @@ OIDC_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" \ --query "cluster.identity.oidc.issuer" --output text)" AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \ --output text) -K8S_VERSION=1.28 +TEMPOUT="$(mktemp)" ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)" AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)" GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)" diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh new file mode 100644 index 000000000000..dcf1cb658dd4 --- /dev/null +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh @@ -0,0 +1,6 @@ +curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \ +&& aws cloudformation deploy \ + --stack-name "Karpenter-${CLUSTER_NAME}" \ + --template-file "${TEMPOUT}" \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameter-overrides "ClusterName=${CLUSTER_NAME}" diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-node-iam.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-node-iam.sh deleted file mode 100644 index 07cd4defce11..000000000000 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step02-node-iam.sh +++ /dev/null @@ -1,15 +0,0 @@ -echo '{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] -}' > node-trust-policy.json - -aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://node-trust-policy.json diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh new file mode 100644 index 000000000000..592e4575d3db --- /dev/null +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh @@ -0,0 +1,26 @@ +cat << EOF > controller-trust-policy.json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", + "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" + } + } + } + ] +} +EOF + +aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --assume-role-policy-document file://controller-trust-policy.json + +aws iam attach-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-node-policies.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-node-policies.sh deleted file mode 100644 index fbc5455e541b..000000000000 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step03-node-policies.sh +++ /dev/null @@ -1,11 +0,0 @@ -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore" diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh deleted file mode 100644 index 2ff658c2f983..000000000000 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh +++ /dev/null @@ -1,142 +0,0 @@ -cat << EOF > controller-trust-policy.json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", - "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" - } - } - } - ] -} -EOF - -aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://controller-trust-policy.json - -cat << EOF > controller-policy.json -{ - "Statement": [ - { - "Action": [ - "ssm:GetParameter", - "ec2:DescribeImages", - "ec2:RunInstances", - "ec2:DescribeSubnets", - "ec2:DescribeSecurityGroups", - "ec2:DescribeLaunchTemplates", - "ec2:DescribeInstances", - "ec2:DescribeInstanceTypes", - "ec2:DescribeInstanceTypeOfferings", - "ec2:DeleteLaunchTemplate", - "ec2:CreateTags", - "ec2:CreateLaunchTemplate", - "ec2:CreateFleet", - "ec2:DescribeSpotPriceHistory", - "pricing:GetProducts" - ], - "Effect": "Allow", - "Resource": "*", - "Sid": "Karpenter" - }, - { - "Action": "ec2:TerminateInstances", - "Condition": { - "StringLike": { - "ec2:ResourceTag/karpenter.sh/nodepool": "*" - } - }, - "Effect": "Allow", - "Resource": "*", - "Sid": "ConditionalEC2Termination" - }, - { - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}", - "Sid": "PassNodeIAMRole" - }, - { - "Effect": "Allow", - "Action": "eks:DescribeCluster", - "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}", - "Sid": "EKSClusterEndpointLookup" - }, - { - "Sid": "AllowScopedInstanceProfileCreationActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:CreateInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileTagActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:TagInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}", - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*", - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:AddRoleToInstanceProfile", - "iam:RemoveRoleFromInstanceProfile", - "iam:DeleteInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowInstanceProfileReadActions", - "Effect": "Allow", - "Resource": "*", - "Action": "iam:GetInstanceProfile" - } - ], - "Version": "2012-10-17" -} -EOF - -aws iam put-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --policy-name "KarpenterControllerPolicy-${CLUSTER_NAME}" \ - --policy-document file://controller-policy.json diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh similarity index 100% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh similarity index 68% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh index c63bde3b78dc..764b3917e3f4 100644 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh @@ -5,13 +5,6 @@ LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ --nodegroup-name "${NODEGROUP}" --query 'nodegroup.launchTemplate.{id:id,version:version}' \ --output text | tr -s "\t" ",") -# If your EKS setup is configured to use only Cluster security group, then please execute - - -SECURITY_GROUPS=$(aws eks describe-cluster \ - --name "${CLUSTER_NAME}" --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text) - -# If your setup uses the security groups in the Launch template of a managed node group, then : - SECURITY_GROUPS="$(aws ec2 describe-launch-template-versions \ --launch-template-id "${LAUNCH_TEMPLATE%,*}" --versions "${LAUNCH_TEMPLATE#*,}" \ --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh similarity index 100% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step07-deploy.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step07-deploy.sh new file mode 100644 index 000000000000..01128a09c884 --- /dev/null +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step07-deploy.sh @@ -0,0 +1,10 @@ +helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set "settings.clusterName=${CLUSTER_NAME}" \ + --set "settings.interruptionQueue=${CLUSTER_NAME}" \ + --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + -f karpenter-node-affinity.yaml \ + --wait diff --git a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh similarity index 87% rename from website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh index 34bf95bdce4c..85213a3457c3 100644 --- a/website/content/en/v0.32/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh +++ b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh @@ -1,5 +1,5 @@ cat < karpenter.yaml diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step09-deploy.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step09-deploy.sh deleted file mode 100644 index e46742fd22ea..000000000000 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step09-deploy.sh +++ /dev/null @@ -1,8 +0,0 @@ -kubectl create namespace "${KARPENTER_NAMESPACE}" || true -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml" -kubectl apply -f karpenter.yaml diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh similarity index 100% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh similarity index 100% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh b/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh similarity index 100% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh rename to website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/_index.md b/website/content/en/v0.37/getting-started/migrating-from-cas/_index.md index 5c04445ba60a..96a20267bbb3 100644 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/_index.md +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/_index.md @@ -10,19 +10,22 @@ This guide will show you how to switch from the [Kubernetes Cluster Autoscaler]( We will make the following assumptions in this guide * You will use an existing EKS cluster -* You will use existing VPC and subnets -* You will use existing security groups +* You will use an existing VPC and subnets +* You will use an existing security groups * Your nodes are part of one or more node groups * Your workloads have pod disruption budgets that adhere to [EKS best practices](https://aws.github.io/aws-eks-best-practices/karpenter/) * Your cluster has an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for service accounts -This guide will also assume you have the `aws` CLI installed. +This guide will also assume you have the `aws` CLI and Helm client installed. You can also perform many of these steps in the console, but we will use the command line for simplicity. -Set a variable for your cluster name. +## Set environment variables +Set the Karpenter and Kubernetes version. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. ```bash -KARPENTER_NAMESPACE=kube-system +KARPENTER_NAMESPACE="kube-system" +KARPENTER_VERSION="{{< param "latest_release_version" >}}" +K8S_VERSION="{{< param "latest_k8s_version" >}}" CLUSTER_NAME= ``` @@ -30,45 +33,68 @@ Set other variables from your cluster configuration. {{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} -Use that information to create our IAM roles, inline policy, and trust relationship. +{{% alert title="Warning" color="warning" %}} +If you open a new shell to run steps in this procedure, you need to set the environment variables again. +{{% /alert %}} ## Create IAM roles -To get started with our migration we first need to create two new IAM roles for nodes provisioned with Karpenter and the Karpenter controller. - -To create the Karpenter node role we will use the following policy and commands. - -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-node-iam.sh" language="bash" %}} +Use CloudFormation to set up the infrastructure needed by the existing EKS cluster. See [CloudFormation]({{< relref "../../reference/cloudformation/" >}}) for a complete description of what `cloudformation.yaml` does for Karpenter. The provided `cloudformation.yaml` template simplifies this setup by creating and configuring all necessary resources, including: -Now attach the required policies to the role + - **IAM Roles and Policies**: Grants Karpenter permissions to interact with EKS, autoscaling, and EC2 services, enabling it to manage nodes dynamically. + - **Instance Profiles**: Attaches necessary permissions to EC2 instances, allowing them to join the cluster and participate in automated scaling as managed by Karpenter. + - **Interruption Queue and Policies**: Setup Amazon SQS queue and Event Rules for handling interruption notifications from AWS services related to EC2 Spot instances and AWS Health events. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-node-policies.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which requires an OIDC endpoint. -If you have another option for using IAM credentials with workloads (e.g. [kube2iam](https://github.com/jtblin/kube2iam)) your steps will be different. +If you have another option for using IAM credentials with workloads (e.g. [Amazon EKS Pod Identity Agent](https://github.com/aws/eks-pod-identity-agent)) your steps will be different. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh" language="bash" %}} + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} ## Add tags to subnets and security groups -We need to add tags to our nodegroup subnets so Karpenter will know which subnets to use. +In order for Karpenter to know which [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to use, we need to add appropriate tags to the nodegroup subnets and security groups. + +### Tag nodegroup subnets -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} -Add tags to our security groups. -This command only tags the security groups for the first nodegroup in the cluster. -If you have multiple nodegroups or multiple security groups you will need to decide which one Karpenter should use. +This loop ensures that Karpenter will be aware of which subnets are associated with each nodegroup by tagging them with karpenter.sh/discovery. + +### Tag security groups + +If your EKS setup is configured to use cluster security group and additional security groups, execute the following commands to tag them for Karpenter discovery: + +```bash +SECURITY_GROUPS=$(aws eks describe-cluster \ + --name "${CLUSTER_NAME}" \ + --query "cluster.resourcesVpcConfig" \ + --output json | jq -r '[.clusterSecurityGroupId] + .securityGroupIds | join(" ")') + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources ${SECURITY_GROUPS} +``` + +If your setup uses security groups from the Launch template of a managed nodegroup, execute the following: + +Note that this command will only tag the security groups for the first nodegroup in the cluster. If you have multiple nodegroups groups, you will need to decide which ones Karpenter should use. + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} + +Alternatively, the subnets and security groups can also be defined in the [NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/) definition by specifying the [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnets) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to be used. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh" language="bash" %}} ## Update aws-auth ConfigMap We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the `aws-auth` ConfigMap in the cluster. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} You will need to add a section to the mapRoles that looks something like this. Replace the `${AWS_PARTITION}` variable with the account partition, `${AWS_ACCOUNT_ID}` variable with your account ID, and `${CLUSTER_NAME}` variable with the cluster name, but do not replace the `{{EC2PrivateDNSName}}`. @@ -88,53 +114,54 @@ The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group. ## Deploy Karpenter +To deploy Karpenter, you can use Helm, which simplifies the installation process by handling Karpenter’s dependencies and configuration files automatically. The Helm command provided below will also incorporate any customized settings, such as node affinity, to align with your specific deployment needs. -First set the Karpenter release you want to deploy. - -```bash -export KARPENTER_VERSION="0.37.0" -``` +### Set Node Affinity for Karpenter -We can now generate a full Karpenter deployment yaml from the Helm chart. +To optimize resource usage and ensure that Karpenter schedules its pods on nodes within a specific, existing node group, it is essential to configure node affinity. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh" language="bash" %}} +Create a file named karpenter-node-affinity.yaml to define the node affinity settings and specify the node group where you want Karpenter to deploy. -Modify the following lines in the karpenter.yaml file. +Be sure to replace `${NODEGROUP}` with the actual name of your node group. -### Set node affinity - -Edit the karpenter.yaml file and find the karpenter deployment affinity rules. -Modify the affinity so karpenter will run on one of the existing node group nodes. - -The rules should look something like this. -Modify the value to match your `$NODEGROUP`, one node group per line. - -```yaml +```bash +cat < karpenter-node-affinity.yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - - matchExpressions: - - key: karpenter.sh/nodepool - operator: DoesNotExist - - key: eks.amazonaws.com/nodegroup - operator: In - values: - - ${NODEGROUP} - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - topologyKey: "kubernetes.io/hostname" + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist + - key: eks.amazonaws.com/nodegroup + operator: In + values: + - "${NODEGROUP}" +EOF ``` -Now that our deployment is ready we can create the karpenter namespace, create the NodePool CRD, and then deploy the rest of the karpenter resources. +Now that you have prepared the node affinity configuration, you can proceed to install Karpenter using Helm. This command includes the affinity settings along with other necessary configurations: -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-deploy.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} + +Expected output: +```bash +Release "karpenter" does not exist. Installing it now. +Pulled: public.ecr.aws/karpenter/karpenter:1.0.5 +Digest: sha256:98382d6406a3c85711269112fbb337c056d4debabaefb936db2d10137b58bd1b +NAME: karpenter +LAST DEPLOYED: Wed Nov 6 16:51:41 2024 +NAMESPACE: kube-system +STATUS: deployed +REVISION: 1 +TEST SUITE: None +``` ## Create default NodePool -We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree/v0.37.2/examples/v1beta1) for specific needs. +We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree{{< githubRelRef >}}examples/v1) for specific needs. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} ## Set nodeAffinity for critical workloads (optional) @@ -165,7 +192,7 @@ affinity: Now that karpenter is running we can disable the cluster autoscaler. To do that we will scale the number of replicas to zero. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh" language="bash" %}} To get rid of the instances that were added from the node group we can scale our nodegroup down to a minimum size to support Karpenter and other critical services. @@ -173,25 +200,26 @@ To get rid of the instances that were added from the node group we can scale our If you have a single multi-AZ node group, we suggest a minimum of 2 instances. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh" language="bash" %}} Or, if you have multiple single-AZ node groups, we suggest a minimum of 1 instance each. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh" language="bash" %}} {{% alert title="Note" color="warning" %}} If you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured. {{% /alert %}} + ## Verify Karpenter As nodegroup nodes are drained you can verify that Karpenter is creating nodes for your workloads. ```bash -kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenter +kubectl logs -f -n $KARPENTER_NAMESPACE -c controller -l app.kubernetes.io/name=karpenter ``` -You should also see new nodes created in your cluster as the old nodes are removed +You should also see new nodes created in your cluster as the old nodes are removed. ```bash kubectl get nodes diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step01-env.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step01-env.sh index b3ee0fa3d0ef..3ee74647cb3c 100644 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step01-env.sh +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step01-env.sh @@ -4,7 +4,7 @@ OIDC_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" \ --query "cluster.identity.oidc.issuer" --output text)" AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \ --output text) -K8S_VERSION=1.28 +TEMPOUT="$(mktemp)" ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)" AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)" GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)" diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh new file mode 100644 index 000000000000..dcf1cb658dd4 --- /dev/null +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh @@ -0,0 +1,6 @@ +curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \ +&& aws cloudformation deploy \ + --stack-name "Karpenter-${CLUSTER_NAME}" \ + --template-file "${TEMPOUT}" \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameter-overrides "ClusterName=${CLUSTER_NAME}" diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-node-iam.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-node-iam.sh deleted file mode 100644 index 07cd4defce11..000000000000 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step02-node-iam.sh +++ /dev/null @@ -1,15 +0,0 @@ -echo '{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] -}' > node-trust-policy.json - -aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://node-trust-policy.json diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh new file mode 100644 index 000000000000..592e4575d3db --- /dev/null +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh @@ -0,0 +1,26 @@ +cat << EOF > controller-trust-policy.json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", + "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" + } + } + } + ] +} +EOF + +aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --assume-role-policy-document file://controller-trust-policy.json + +aws iam attach-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-node-policies.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-node-policies.sh deleted file mode 100644 index fbc5455e541b..000000000000 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step03-node-policies.sh +++ /dev/null @@ -1,11 +0,0 @@ -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore" diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh deleted file mode 100644 index 2ff658c2f983..000000000000 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh +++ /dev/null @@ -1,142 +0,0 @@ -cat << EOF > controller-trust-policy.json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", - "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" - } - } - } - ] -} -EOF - -aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://controller-trust-policy.json - -cat << EOF > controller-policy.json -{ - "Statement": [ - { - "Action": [ - "ssm:GetParameter", - "ec2:DescribeImages", - "ec2:RunInstances", - "ec2:DescribeSubnets", - "ec2:DescribeSecurityGroups", - "ec2:DescribeLaunchTemplates", - "ec2:DescribeInstances", - "ec2:DescribeInstanceTypes", - "ec2:DescribeInstanceTypeOfferings", - "ec2:DeleteLaunchTemplate", - "ec2:CreateTags", - "ec2:CreateLaunchTemplate", - "ec2:CreateFleet", - "ec2:DescribeSpotPriceHistory", - "pricing:GetProducts" - ], - "Effect": "Allow", - "Resource": "*", - "Sid": "Karpenter" - }, - { - "Action": "ec2:TerminateInstances", - "Condition": { - "StringLike": { - "ec2:ResourceTag/karpenter.sh/nodepool": "*" - } - }, - "Effect": "Allow", - "Resource": "*", - "Sid": "ConditionalEC2Termination" - }, - { - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}", - "Sid": "PassNodeIAMRole" - }, - { - "Effect": "Allow", - "Action": "eks:DescribeCluster", - "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}", - "Sid": "EKSClusterEndpointLookup" - }, - { - "Sid": "AllowScopedInstanceProfileCreationActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:CreateInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileTagActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:TagInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}", - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*", - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:AddRoleToInstanceProfile", - "iam:RemoveRoleFromInstanceProfile", - "iam:DeleteInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowInstanceProfileReadActions", - "Effect": "Allow", - "Resource": "*", - "Action": "iam:GetInstanceProfile" - } - ], - "Version": "2012-10-17" -} -EOF - -aws iam put-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --policy-name "KarpenterControllerPolicy-${CLUSTER_NAME}" \ - --policy-document file://controller-policy.json diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh similarity index 68% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh index c63bde3b78dc..764b3917e3f4 100644 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh @@ -5,13 +5,6 @@ LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ --nodegroup-name "${NODEGROUP}" --query 'nodegroup.launchTemplate.{id:id,version:version}' \ --output text | tr -s "\t" ",") -# If your EKS setup is configured to use only Cluster security group, then please execute - - -SECURITY_GROUPS=$(aws eks describe-cluster \ - --name "${CLUSTER_NAME}" --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text) - -# If your setup uses the security groups in the Launch template of a managed node group, then : - SECURITY_GROUPS="$(aws ec2 describe-launch-template-versions \ --launch-template-id "${LAUNCH_TEMPLATE%,*}" --versions "${LAUNCH_TEMPLATE#*,}" \ --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh similarity index 100% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step07-deploy.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step07-deploy.sh new file mode 100644 index 000000000000..01128a09c884 --- /dev/null +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step07-deploy.sh @@ -0,0 +1,10 @@ +helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set "settings.clusterName=${CLUSTER_NAME}" \ + --set "settings.interruptionQueue=${CLUSTER_NAME}" \ + --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + -f karpenter-node-affinity.yaml \ + --wait diff --git a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh similarity index 87% rename from website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh index 33f1cb553b1b..85213a3457c3 100644 --- a/website/content/en/v0.36/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh +++ b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh @@ -1,5 +1,5 @@ cat < karpenter.yaml diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step09-deploy.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step09-deploy.sh deleted file mode 100644 index e46742fd22ea..000000000000 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step09-deploy.sh +++ /dev/null @@ -1,8 +0,0 @@ -kubectl create namespace "${KARPENTER_NAMESPACE}" || true -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml" -kubectl apply -f karpenter.yaml diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh similarity index 100% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh b/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh rename to website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/_index.md b/website/content/en/v1.0/getting-started/migrating-from-cas/_index.md index f37df96642c1..96a20267bbb3 100644 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/_index.md +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/_index.md @@ -10,19 +10,22 @@ This guide will show you how to switch from the [Kubernetes Cluster Autoscaler]( We will make the following assumptions in this guide * You will use an existing EKS cluster -* You will use existing VPC and subnets -* You will use existing security groups +* You will use an existing VPC and subnets +* You will use an existing security groups * Your nodes are part of one or more node groups * Your workloads have pod disruption budgets that adhere to [EKS best practices](https://aws.github.io/aws-eks-best-practices/karpenter/) * Your cluster has an [OIDC provider](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) for service accounts -This guide will also assume you have the `aws` CLI installed. +This guide will also assume you have the `aws` CLI and Helm client installed. You can also perform many of these steps in the console, but we will use the command line for simplicity. -Set a variable for your cluster name. +## Set environment variables +Set the Karpenter and Kubernetes version. Check the [Compatibility Matrix](https://karpenter.sh/docs/upgrading/compatibility/) to find the Karpenter version compatible with your current Amazon EKS version. ```bash -KARPENTER_NAMESPACE=kube-system +KARPENTER_NAMESPACE="kube-system" +KARPENTER_VERSION="{{< param "latest_release_version" >}}" +K8S_VERSION="{{< param "latest_k8s_version" >}}" CLUSTER_NAME= ``` @@ -30,45 +33,68 @@ Set other variables from your cluster configuration. {{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step01-env.sh" language="bash" %}} -Use that information to create our IAM roles, inline policy, and trust relationship. +{{% alert title="Warning" color="warning" %}} +If you open a new shell to run steps in this procedure, you need to set the environment variables again. +{{% /alert %}} ## Create IAM roles -To get started with our migration we first need to create two new IAM roles for nodes provisioned with Karpenter and the Karpenter controller. - -To create the Karpenter node role we will use the following policy and commands. - -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-node-iam.sh" language="bash" %}} +Use CloudFormation to set up the infrastructure needed by the existing EKS cluster. See [CloudFormation]({{< relref "../../reference/cloudformation/" >}}) for a complete description of what `cloudformation.yaml` does for Karpenter. The provided `cloudformation.yaml` template simplifies this setup by creating and configuring all necessary resources, including: -Now attach the required policies to the role + - **IAM Roles and Policies**: Grants Karpenter permissions to interact with EKS, autoscaling, and EC2 services, enabling it to manage nodes dynamically. + - **Instance Profiles**: Attaches necessary permissions to EC2 instances, allowing them to join the cluster and participate in automated scaling as managed by Karpenter. + - **Interruption Queue and Policies**: Setup Amazon SQS queue and Event Rules for handling interruption notifications from AWS services related to EC2 Spot instances and AWS Health events. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-node-policies.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh" language="bash" %}} Now we need to create an IAM role that the Karpenter controller will use to provision new instances. The controller will be using [IAM Roles for Service Accounts (IRSA)](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) which requires an OIDC endpoint. -If you have another option for using IAM credentials with workloads (e.g. [kube2iam](https://github.com/jtblin/kube2iam)) your steps will be different. +If you have another option for using IAM credentials with workloads (e.g. [Amazon EKS Pod Identity Agent](https://github.com/aws/eks-pod-identity-agent)) your steps will be different. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh" language="bash" %}} + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh" language="bash" %}} ## Add tags to subnets and security groups -We need to add tags to our nodegroup subnets so Karpenter will know which subnets to use. +In order for Karpenter to know which [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to use, we need to add appropriate tags to the nodegroup subnets and security groups. + +### Tag nodegroup subnets -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-subnets.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh" language="bash" %}} -Add tags to our security groups. -This command only tags the security groups for the first nodegroup in the cluster. -If you have multiple nodegroups or multiple security groups you will need to decide which one Karpenter should use. +This loop ensures that Karpenter will be aware of which subnets are associated with each nodegroup by tagging them with karpenter.sh/discovery. + +### Tag security groups + +If your EKS setup is configured to use cluster security group and additional security groups, execute the following commands to tag them for Karpenter discovery: + +```bash +SECURITY_GROUPS=$(aws eks describe-cluster \ + --name "${CLUSTER_NAME}" \ + --query "cluster.resourcesVpcConfig" \ + --output json | jq -r '[.clusterSecurityGroupId] + .securityGroupIds | join(" ")') + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources ${SECURITY_GROUPS} +``` + +If your setup uses security groups from the Launch template of a managed nodegroup, execute the following: + +Note that this command will only tag the security groups for the first nodegroup in the cluster. If you have multiple nodegroups groups, you will need to decide which ones Karpenter should use. + +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh" language="bash" %}} + +Alternatively, the subnets and security groups can also be defined in the [NodeClasses](https://karpenter.sh/docs/concepts/nodeclasses/) definition by specifying the [subnets](https://karpenter.sh/docs/concepts/nodeclasses/#specsubnets) and [security groups](https://karpenter.sh/docs/concepts/nodeclasses/#specsecuritygroupselectorterms) to be used. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-tag-security-groups.sh" language="bash" %}} ## Update aws-auth ConfigMap We need to allow nodes that are using the node IAM role we just created to join the cluster. To do that we have to modify the `aws-auth` ConfigMap in the cluster. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh" language="bash" %}} You will need to add a section to the mapRoles that looks something like this. Replace the `${AWS_PARTITION}` variable with the account partition, `${AWS_ACCOUNT_ID}` variable with your account ID, and `${CLUSTER_NAME}` variable with the cluster name, but do not replace the `{{EC2PrivateDNSName}}`. @@ -88,53 +114,54 @@ The full aws-auth configmap should have two groups. One for your Karpenter node role and one for your existing node group. ## Deploy Karpenter +To deploy Karpenter, you can use Helm, which simplifies the installation process by handling Karpenter’s dependencies and configuration files automatically. The Helm command provided below will also incorporate any customized settings, such as node affinity, to align with your specific deployment needs. -First set the Karpenter release you want to deploy. - -```bash -export KARPENTER_VERSION="1.0.7" -``` +### Set Node Affinity for Karpenter -We can now generate a full Karpenter deployment yaml from the Helm chart. +To optimize resource usage and ensure that Karpenter schedules its pods on nodes within a specific, existing node group, it is essential to configure node affinity. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-generate-chart.sh" language="bash" %}} +Create a file named karpenter-node-affinity.yaml to define the node affinity settings and specify the node group where you want Karpenter to deploy. -Modify the following lines in the karpenter.yaml file. +Be sure to replace `${NODEGROUP}` with the actual name of your node group. -### Set node affinity - -Edit the karpenter.yaml file and find the karpenter deployment affinity rules. -Modify the affinity so karpenter will run on one of the existing node group nodes. - -The rules should look something like this. -Modify the value to match your `$NODEGROUP`, one node group per line. - -```yaml +```bash +cat < karpenter-node-affinity.yaml affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - - matchExpressions: - - key: karpenter.sh/nodepool - operator: DoesNotExist - - key: eks.amazonaws.com/nodegroup - operator: In - values: - - ${NODEGROUP} - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - topologyKey: "kubernetes.io/hostname" + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist + - key: eks.amazonaws.com/nodegroup + operator: In + values: + - "${NODEGROUP}" +EOF ``` -Now that our deployment is ready we can create the karpenter namespace, create the NodePool CRD, and then deploy the rest of the karpenter resources. +Now that you have prepared the node affinity configuration, you can proceed to install Karpenter using Helm. This command includes the affinity settings along with other necessary configurations: -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-deploy.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step07-deploy.sh" language="bash" %}} + +Expected output: +```bash +Release "karpenter" does not exist. Installing it now. +Pulled: public.ecr.aws/karpenter/karpenter:1.0.5 +Digest: sha256:98382d6406a3c85711269112fbb337c056d4debabaefb936db2d10137b58bd1b +NAME: karpenter +LAST DEPLOYED: Wed Nov 6 16:51:41 2024 +NAMESPACE: kube-system +STATUS: deployed +REVISION: 1 +TEST SUITE: None +``` ## Create default NodePool -We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree/v1.0.7/examples/v1) for specific needs. +We need to create a default NodePool so Karpenter knows what types of nodes we want for unscheduled workloads. You can refer to some of the [example NodePool](https://github.com/aws/karpenter/tree{{< githubRelRef >}}examples/v1) for specific needs. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh" language="bash" %}} ## Set nodeAffinity for critical workloads (optional) @@ -165,7 +192,7 @@ affinity: Now that karpenter is running we can disable the cluster autoscaler. To do that we will scale the number of replicas to zero. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh" language="bash" %}} To get rid of the instances that were added from the node group we can scale our nodegroup down to a minimum size to support Karpenter and other critical services. @@ -173,25 +200,26 @@ To get rid of the instances that were added from the node group we can scale our If you have a single multi-AZ node group, we suggest a minimum of 2 instances. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-single-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh" language="bash" %}} Or, if you have multiple single-AZ node groups, we suggest a minimum of 1 instance each. -{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step12-scale-multiple-ng.sh" language="bash" %}} +{{% script file="./content/en/{VERSION}/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh" language="bash" %}} {{% alert title="Note" color="warning" %}} If you have a lot of nodes or workloads you may want to slowly scale down your node groups by a few instances at a time. It is recommended to watch the transition carefully for workloads that may not have enough replicas running or disruption budgets configured. {{% /alert %}} + ## Verify Karpenter As nodegroup nodes are drained you can verify that Karpenter is creating nodes for your workloads. ```bash -kubectl logs -f -n karpenter -c controller -l app.kubernetes.io/name=karpenter +kubectl logs -f -n $KARPENTER_NAMESPACE -c controller -l app.kubernetes.io/name=karpenter ``` -You should also see new nodes created in your cluster as the old nodes are removed +You should also see new nodes created in your cluster as the old nodes are removed. ```bash kubectl get nodes diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step01-env.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step01-env.sh index b3ee0fa3d0ef..3ee74647cb3c 100644 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step01-env.sh +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step01-env.sh @@ -4,7 +4,7 @@ OIDC_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" \ --query "cluster.identity.oidc.issuer" --output text)" AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \ --output text) -K8S_VERSION=1.28 +TEMPOUT="$(mktemp)" ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)" AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)" GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh new file mode 100644 index 000000000000..dcf1cb658dd4 --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-cloudformation-setup.sh @@ -0,0 +1,6 @@ +curl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \ +&& aws cloudformation deploy \ + --stack-name "Karpenter-${CLUSTER_NAME}" \ + --template-file "${TEMPOUT}" \ + --capabilities CAPABILITY_NAMED_IAM \ + --parameter-overrides "ClusterName=${CLUSTER_NAME}" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-node-iam.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-node-iam.sh deleted file mode 100644 index 07cd4defce11..000000000000 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step02-node-iam.sh +++ /dev/null @@ -1,15 +0,0 @@ -echo '{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Service": "ec2.amazonaws.com" - }, - "Action": "sts:AssumeRole" - } - ] -}' > node-trust-policy.json - -aws iam create-role --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://node-trust-policy.json diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh new file mode 100644 index 000000000000..592e4575d3db --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-controller-iam.sh @@ -0,0 +1,26 @@ +cat << EOF > controller-trust-policy.json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { + "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { + "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", + "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" + } + } + } + ] +} +EOF + +aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --assume-role-policy-document file://controller-trust-policy.json + +aws iam attach-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ + --policy-arn "arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-node-policies.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-node-policies.sh deleted file mode 100644 index fbc5455e541b..000000000000 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step03-node-policies.sh +++ /dev/null @@ -1,11 +0,0 @@ -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKSWorkerNodePolicy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEKS_CNI_Policy" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly" - -aws iam attach-role-policy --role-name "KarpenterNodeRole-${CLUSTER_NAME}" \ - --policy-arn "arn:${AWS_PARTITION}:iam::aws:policy/AmazonSSMManagedInstanceCore" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh deleted file mode 100644 index 2ff658c2f983..000000000000 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-controller-iam.sh +++ /dev/null @@ -1,142 +0,0 @@ -cat << EOF > controller-trust-policy.json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Principal": { - "Federated": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_ENDPOINT#*//}" - }, - "Action": "sts:AssumeRoleWithWebIdentity", - "Condition": { - "StringEquals": { - "${OIDC_ENDPOINT#*//}:aud": "sts.amazonaws.com", - "${OIDC_ENDPOINT#*//}:sub": "system:serviceaccount:${KARPENTER_NAMESPACE}:karpenter" - } - } - } - ] -} -EOF - -aws iam create-role --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --assume-role-policy-document file://controller-trust-policy.json - -cat << EOF > controller-policy.json -{ - "Statement": [ - { - "Action": [ - "ssm:GetParameter", - "ec2:DescribeImages", - "ec2:RunInstances", - "ec2:DescribeSubnets", - "ec2:DescribeSecurityGroups", - "ec2:DescribeLaunchTemplates", - "ec2:DescribeInstances", - "ec2:DescribeInstanceTypes", - "ec2:DescribeInstanceTypeOfferings", - "ec2:DeleteLaunchTemplate", - "ec2:CreateTags", - "ec2:CreateLaunchTemplate", - "ec2:CreateFleet", - "ec2:DescribeSpotPriceHistory", - "pricing:GetProducts" - ], - "Effect": "Allow", - "Resource": "*", - "Sid": "Karpenter" - }, - { - "Action": "ec2:TerminateInstances", - "Condition": { - "StringLike": { - "ec2:ResourceTag/karpenter.sh/nodepool": "*" - } - }, - "Effect": "Allow", - "Resource": "*", - "Sid": "ConditionalEC2Termination" - }, - { - "Effect": "Allow", - "Action": "iam:PassRole", - "Resource": "arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}", - "Sid": "PassNodeIAMRole" - }, - { - "Effect": "Allow", - "Action": "eks:DescribeCluster", - "Resource": "arn:${AWS_PARTITION}:eks:${AWS_REGION}:${AWS_ACCOUNT_ID}:cluster/${CLUSTER_NAME}", - "Sid": "EKSClusterEndpointLookup" - }, - { - "Sid": "AllowScopedInstanceProfileCreationActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:CreateInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileTagActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:TagInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}", - "aws:RequestTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:RequestTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*", - "aws:RequestTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowScopedInstanceProfileActions", - "Effect": "Allow", - "Resource": "*", - "Action": [ - "iam:AddRoleToInstanceProfile", - "iam:RemoveRoleFromInstanceProfile", - "iam:DeleteInstanceProfile" - ], - "Condition": { - "StringEquals": { - "aws:ResourceTag/kubernetes.io/cluster/${CLUSTER_NAME}": "owned", - "aws:ResourceTag/topology.kubernetes.io/region": "${AWS_REGION}" - }, - "StringLike": { - "aws:ResourceTag/karpenter.k8s.aws/ec2nodeclass": "*" - } - } - }, - { - "Sid": "AllowInstanceProfileReadActions", - "Effect": "Allow", - "Resource": "*", - "Action": "iam:GetInstanceProfile" - } - ], - "Version": "2012-10-17" -} -EOF - -aws iam put-role-policy --role-name "KarpenterControllerRole-${CLUSTER_NAME}" \ - --policy-name "KarpenterControllerPolicy-${CLUSTER_NAME}" \ - --policy-document file://controller-policy.json diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh new file mode 100644 index 000000000000..31e8093cd6f8 --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step04-tag-subnets.sh @@ -0,0 +1,6 @@ +for NODEGROUP in $(aws eks list-nodegroups --cluster-name "${CLUSTER_NAME}" --query 'nodegroups' --output text); do + aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources $(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ + --nodegroup-name "${NODEGROUP}" --query 'nodegroup.subnets' --output text ) +done diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh new file mode 100644 index 000000000000..764b3917e3f4 --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step05-tag-security-groups.sh @@ -0,0 +1,15 @@ +NODEGROUP=$(aws eks list-nodegroups --cluster-name "${CLUSTER_NAME}" \ + --query 'nodegroups[0]' --output text) + +LAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name "${CLUSTER_NAME}" \ + --nodegroup-name "${NODEGROUP}" --query 'nodegroup.launchTemplate.{id:id,version:version}' \ + --output text | tr -s "\t" ",") + +SECURITY_GROUPS="$(aws ec2 describe-launch-template-versions \ + --launch-template-id "${LAUNCH_TEMPLATE%,*}" --versions "${LAUNCH_TEMPLATE#*,}" \ + --query 'LaunchTemplateVersions[0].LaunchTemplateData.[NetworkInterfaces[0].Groups||SecurityGroupIds]' \ + --output text)" + +aws ec2 create-tags \ + --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \ + --resources "${SECURITY_GROUPS}" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step07-edit-aws-auth.sh rename to website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step06-edit-aws-auth.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step07-deploy.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step07-deploy.sh new file mode 100644 index 000000000000..01128a09c884 --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step07-deploy.sh @@ -0,0 +1,10 @@ +helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set "settings.clusterName=${CLUSTER_NAME}" \ + --set "settings.interruptionQueue=${CLUSTER_NAME}" \ + --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + -f karpenter-node-affinity.yaml \ + --wait diff --git a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh similarity index 87% rename from website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh rename to website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh index 33f1cb553b1b..85213a3457c3 100644 --- a/website/content/en/v0.37/getting-started/migrating-from-cas/scripts/step10-create-nodepool.sh +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step08-create-nodepool.sh @@ -1,5 +1,5 @@ cat < karpenter.yaml diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step09-deploy.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step09-deploy.sh deleted file mode 100644 index e46742fd22ea..000000000000 --- a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step09-deploy.sh +++ /dev/null @@ -1,8 +0,0 @@ -kubectl create namespace "${KARPENTER_NAMESPACE}" || true -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml" -kubectl create -f \ - "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml" -kubectl apply -f karpenter.yaml diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh similarity index 100% rename from website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step11-scale-cas.sh rename to website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step09-scale-cas.sh diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh new file mode 100644 index 000000000000..efce68394779 --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step10-scale-single-ng.sh @@ -0,0 +1,3 @@ +aws eks update-nodegroup-config --cluster-name "${CLUSTER_NAME}" \ + --nodegroup-name "${NODEGROUP}" \ + --scaling-config "minSize=2,maxSize=2,desiredSize=2" diff --git a/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh new file mode 100644 index 000000000000..a28c9759bcdb --- /dev/null +++ b/website/content/en/v1.0/getting-started/migrating-from-cas/scripts/step11-scale-multiple-ng.sh @@ -0,0 +1,5 @@ +for NODEGROUP in $(aws eks list-nodegroups --cluster-name "${CLUSTER_NAME}" \ + --query 'nodegroups' --output text); do aws eks update-nodegroup-config --cluster-name "${CLUSTER_NAME}" \ + --nodegroup-name "${NODEGROUP}" \ + --scaling-config "minSize=1,maxSize=1,desiredSize=1" +done