From abcc9f4e7fdacfb78e46ac40c86b6af630a96111 Mon Sep 17 00:00:00 2001 From: Amanuel Engeda Date: Sun, 11 Aug 2024 16:52:57 -0700 Subject: [PATCH] Update docs to allow webhooks to enabled by the helm controller --- .../en/preview/upgrading/v1-migration.md | 60 ++++++++++++++++--- 1 file changed, 52 insertions(+), 8 deletions(-) diff --git a/website/content/en/preview/upgrading/v1-migration.md b/website/content/en/preview/upgrading/v1-migration.md index ee485ed12c52..db9058564c63 100644 --- a/website/content/en/preview/upgrading/v1-migration.md +++ b/website/content/en/preview/upgrading/v1-migration.md @@ -44,18 +44,59 @@ Please read through the entire procedure before beginning the upgrade. There are 2. Review for breaking changes between v0.33 and v0.37: If you are already running Karpenter v0.37.x, you can skip this step. If you are running an earlier Karpenter version, you need to review the [Upgrade Guide]({{}}) for each minor release. -3. Set environment variables for your cluster: +3. Set environment variables for your cluster to upgrade to the latest patch version of the current Karpenter version you're running on: ```bash export KARPENTER_NAMESPACE=kube-system - export KARPENTER_VERSION=1.0.0 + export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter" export AWS_PARTITION="aws" # if you are not using standard partitions, you may need to configure to aws-cn / aws-us-gov export CLUSTER_NAME="${USER}-karpenter-demo" export AWS_REGION="us-west-2" export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)" ``` -4. Update your existing policy using the following: +4. Set environment variables for upgrading to the latest patch version: + + ```bash + export KARPENTER_VERSION= + ``` + +6. Apply the latest patch version of your current minor version's Custom Resource Definitions (CRDs): + + ```bash + helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set webhook.enabled=true \ + --set webhook.serviceName=karpenter \ + --set webhook.serviceNamespace="${KARPENTER_NAMESPACE}" \ + --set webhook.port=8443 + ``` + + +7. Upgrade Karpenter to the latest patch version of your current minor version's. At the end of this step, conversion webhooks will run but will not convert any version. + + ```bash + # Service account annotation can be dropped when using pod identity + helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ + --set settings.clusterName=${CLUSTER_NAME} \ + --set settings.interruptionQueue=${CLUSTER_NAME} \ + --set controller.resources.requests.cpu=1 \ + --set controller.resources.requests.memory=1Gi \ + --set controller.resources.limits.cpu=1 \ + --set controller.resources.limits.memory=1Gi \ + --set webhook.enabled=true \ + --set webhook.port=8443 \ + --wait + ``` + +8. Set environment variables for first upgrading to v1.0.0 + + ```bash + export KARPENTER_VERSION=1.0.0 + ``` + + +9. Update your existing policy using the following to the v1.0.0 controller policy: Notable Changes to the IAM Policy include `SSM:GetParameter` to `SSM:GetParametersByPath` and additional tag-scoping for the `eks:eks-cluster-name` tag for instances and instance profiles. ```bash @@ -68,20 +109,21 @@ Please read through the entire procedure before beginning the upgrade. There are --parameter-overrides "ClusterName=${CLUSTER_NAME}" ``` -5. Apply the v1.0.0 Custom Resource Definitions (CRDs): +10. Apply the v1.0.0 Custom Resource Definitions (CRDs): - ```bash - helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + ```bash + helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-crd --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set webhook.enabled=true \ --set webhook.serviceName=karpenter \ --set webhook.serviceNamespace="${KARPENTER_NAMESPACE}" \ --set webhook.port=8443 ``` -6. Upgrade Karpenter to the new version. At the end of this step, conversion webhooks run to convert the Karpenter CRDs to v1. +11. Upgrade Karpenter to the new version. At the end of this step, conversion webhooks run to convert the Karpenter CRDs to v1. ```bash helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ + --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ # Service account annotion can be droped when using pod identity --set settings.clusterName=${CLUSTER_NAME} \ --set settings.interruptionQueue=${CLUSTER_NAME} \ --set controller.resources.requests.cpu=1 \ @@ -95,7 +137,7 @@ Please read through the entire procedure before beginning the upgrade. There are Karpenter has deprecated and moved a number of Helm values as part of the v1 release. Ensure that you upgrade to the newer version of these helm values during your migration to v1. You can find detail for all the settings that were moved in the [v1 Upgrade Reference]({{}}). {{% /alert %}} -7. Once upgraded, you won't need to roll your nodes to be compatible with v1.1.0, except if you have multiple NodePools with different `kubelet`s that are referencing the same EC2NodeClass. Karpenter has moved the `kubelet` to the EC2NodeClass in v1. NodePools with different `kubelet` referencing the same EC2NodeClass will be compatible with v1.0.0, but will not be in v1.1.0. +12. Once upgraded, you won't need to roll your nodes to be compatible with v1.1.0, except if you have multiple NodePools with different `kubelet`s that are referencing the same EC2NodeClass. Karpenter has moved the `kubelet` to the EC2NodeClass in v1. NodePools with different `kubelet` referencing the same EC2NodeClass will be compatible with v1.0.0, but will not be in v1.1.0. When you have completed the migration to `1.0.0` CRDs, Karpenter will be able to serve both the `v1beta1` versions and the `v1` versions of NodePools, NodeClaims, and EC2NodeClasses. The results of upgrading these CRDs include the following: @@ -211,6 +253,7 @@ Since both v1beta1 and v1 will be served, `kubectl` will default to returning th ```bash export KARPENTER_NAMESPACE="kube-system" export KARPENTER_VERSION="" +export KARPENTER_IAM_ROLE_ARN="arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/${CLUSTER_NAME}-karpenter" export CLUSTER_NAME="" export TEMPOUT="$(mktemp)" ``` @@ -249,6 +292,7 @@ helm upgrade --install karpenter-crd oci://public.ecr.aws/karpenter/karpenter-cr 4. Rollback the Karpenter Controller ```bash +# Service account annotation can be dropped when using pod identity helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version ${KARPENTER_VERSION} --namespace "${KARPENTER_NAMESPACE}" --create-namespace \ --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=${KARPENTER_IAM_ROLE_ARN} \ --set settings.clusterName=${CLUSTER_NAME} \