diff --git a/website/content/en/docs/concepts/scheduling.md b/website/content/en/docs/concepts/scheduling.md index 2ef5b2c62897..cd6620ce1502 100755 --- a/website/content/en/docs/concepts/scheduling.md +++ b/website/content/en/docs/concepts/scheduling.md @@ -103,6 +103,9 @@ Refer to general [Kubernetes GPU](https://kubernetes.io/docs/tasks/manage-gpus/s {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/docs/reference/settings.md b/website/content/en/docs/reference/settings.md index 4eef61fb3952..150a62c9662b 100644 --- a/website/content/en/docs/reference/settings.md +++ b/website/content/en/docs/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_OUTPUT_PATHS | \-\-log-output-paths | Optional comma separated paths for directing log output (default = stdout)| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8080)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075)| | WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| | WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| diff --git a/website/content/en/docs/troubleshooting.md b/website/content/en/docs/troubleshooting.md index 6819a794bd4c..1f0a5c0cfca2 100644 --- a/website/content/en/docs/troubleshooting.md +++ b/website/content/en/docs/troubleshooting.md @@ -248,6 +248,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your NodePods 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}). diff --git a/website/content/en/preview/concepts/scheduling.md b/website/content/en/preview/concepts/scheduling.md index e18d566407cb..ff81bc3c8c4b 100755 --- a/website/content/en/preview/concepts/scheduling.md +++ b/website/content/en/preview/concepts/scheduling.md @@ -112,6 +112,9 @@ However, Karpenter is not aware of the decisions made by the Neuron scheduler ex {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/preview/reference/settings.md b/website/content/en/preview/reference/settings.md index e62984e05ac3..b8f38ef3afbd 100644 --- a/website/content/en/preview/reference/settings.md +++ b/website/content/en/preview/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_OUTPUT_PATHS | \-\-log-output-paths | Optional comma separated paths for directing log output (default = stdout)| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8080)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types when cached information is unavailable. (default = 0.075, which equals to 7.5%) | [comment]: <> (end docs generated content from hack/docs/configuration_gen_docs.go) diff --git a/website/content/en/preview/troubleshooting.md b/website/content/en/preview/troubleshooting.md index 7589d4ff503e..85e5c1c40599 100644 --- a/website/content/en/preview/troubleshooting.md +++ b/website/content/en/preview/troubleshooting.md @@ -248,6 +248,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your NodePods 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}). diff --git a/website/content/en/v0.32/concepts/scheduling.md b/website/content/en/v0.32/concepts/scheduling.md index 147f4e00896e..b4eeae66b597 100755 --- a/website/content/en/v0.32/concepts/scheduling.md +++ b/website/content/en/v0.32/concepts/scheduling.md @@ -103,6 +103,9 @@ Refer to general [Kubernetes GPU](https://kubernetes.io/docs/tasks/manage-gpus/s {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/v0.32/reference/settings.md b/website/content/en/v0.32/reference/settings.md index a3dc01c41e3d..8c6f2440ac05 100644 --- a/website/content/en/v0.32/reference/settings.md +++ b/website/content/en/v0.32/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_LEVEL | \-\-log-level | Log verbosity level. Can be one of 'debug', 'info', or 'error'| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8000)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075, which equals to 7.5%)| | WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| | WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| diff --git a/website/content/en/v0.32/troubleshooting.md b/website/content/en/v0.32/troubleshooting.md index cbdc94dbac41..177043b3c246 100644 --- a/website/content/en/v0.32/troubleshooting.md +++ b/website/content/en/v0.32/troubleshooting.md @@ -331,6 +331,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your Provisioner 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}). diff --git a/website/content/en/v0.36/concepts/scheduling.md b/website/content/en/v0.36/concepts/scheduling.md index c942ab3b3ef7..d0973d44c67c 100755 --- a/website/content/en/v0.36/concepts/scheduling.md +++ b/website/content/en/v0.36/concepts/scheduling.md @@ -103,6 +103,9 @@ Refer to general [Kubernetes GPU](https://kubernetes.io/docs/tasks/manage-gpus/s {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/v0.36/reference/settings.md b/website/content/en/v0.36/reference/settings.md index 207cd15c4bda..c39b46e50e02 100644 --- a/website/content/en/v0.36/reference/settings.md +++ b/website/content/en/v0.36/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_LEVEL | \-\-log-level | Log verbosity level. Can be one of 'debug', 'info', or 'error' (default = info)| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8000)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075, which equals to 7.5%)| | WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| | WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| diff --git a/website/content/en/v0.36/troubleshooting.md b/website/content/en/v0.36/troubleshooting.md index 461a01a4e6fa..11e76125e924 100644 --- a/website/content/en/v0.36/troubleshooting.md +++ b/website/content/en/v0.36/troubleshooting.md @@ -343,6 +343,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your Provisioner 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}). diff --git a/website/content/en/v0.37/concepts/scheduling.md b/website/content/en/v0.37/concepts/scheduling.md index 437a6f1cda5f..bc85190ead1f 100755 --- a/website/content/en/v0.37/concepts/scheduling.md +++ b/website/content/en/v0.37/concepts/scheduling.md @@ -103,6 +103,9 @@ Refer to general [Kubernetes GPU](https://kubernetes.io/docs/tasks/manage-gpus/s {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/v0.37/reference/settings.md b/website/content/en/v0.37/reference/settings.md index 129882b332d9..6a9bd1a2dfec 100644 --- a/website/content/en/v0.37/reference/settings.md +++ b/website/content/en/v0.37/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_LEVEL | \-\-log-level | Log verbosity level. Can be one of 'debug', 'info', or 'error' (default = info)| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8000)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075, which equals to 7.5%)| | WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| | WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| diff --git a/website/content/en/v0.37/troubleshooting.md b/website/content/en/v0.37/troubleshooting.md index 461a01a4e6fa..11e76125e924 100644 --- a/website/content/en/v0.37/troubleshooting.md +++ b/website/content/en/v0.37/troubleshooting.md @@ -343,6 +343,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your Provisioner 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}). diff --git a/website/content/en/v1.0/concepts/scheduling.md b/website/content/en/v1.0/concepts/scheduling.md index 2ef5b2c62897..8c8b6ae86a88 100755 --- a/website/content/en/v1.0/concepts/scheduling.md +++ b/website/content/en/v1.0/concepts/scheduling.md @@ -103,6 +103,10 @@ Refer to general [Kubernetes GPU](https://kubernetes.io/docs/tasks/manage-gpus/s {{% alert title="Note" color="primary" %}} You must enable Pod ENI support in the AWS VPC CNI Plugin before enabling Pod ENI support in Karpenter. Please refer to the [Security Groups for Pods documentation](https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html) for instructions. {{% /alert %}} +{{% alert title="Note" color="primary" %}} +If you've enabled [Security Groups per Pod](https://aws.github.io/aws-eks-best-practices/networking/sgpp/), one of the instance's ENIs is reserved. To avoid discrepancies between the `maxPods` value and the node's supported pod density, you need to set [RESERVED_ENIS]({{}})=1. +{{% /alert %}} + Here is an example of a pod-eni resource defined in a deployment manifest: ``` diff --git a/website/content/en/v1.0/reference/settings.md b/website/content/en/v1.0/reference/settings.md index 08d483001a73..11f14ac51462 100644 --- a/website/content/en/v1.0/reference/settings.md +++ b/website/content/en/v1.0/reference/settings.md @@ -32,7 +32,7 @@ Karpenter surfaces environment variables and CLI parameters to allow you to conf | LOG_OUTPUT_PATHS | \-\-log-output-paths | Optional comma separated paths for directing log output (default = stdout)| | MEMORY_LIMIT | \-\-memory-limit | Memory limit on the container running the controller. The GC soft memory limit is set to 90% of this value. (default = -1)| | METRICS_PORT | \-\-metrics-port | The port the metric endpoint binds to for operating metrics about the controller itself (default = 8080)| -| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html. (default = 0)| +| RESERVED_ENIS | \-\-reserved-enis | Reserved ENIs are not included in the calculations for max-pods or kube-reserved. This is most often used with the [VPC CNI custom networking setup](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) and [Security Groups for Pods]({{}}). (default = 0)| | VM_MEMORY_OVERHEAD_PERCENT | \-\-vm-memory-overhead-percent | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types. (default = 0.075, which equals to 7.5%)| | WEBHOOK_METRICS_PORT | \-\-webhook-metrics-port | The port the webhook metric endpoing binds to for operating metrics about the webhook (default = 8001)| | WEBHOOK_PORT | \-\-webhook-port | The port the webhook endpoint binds to for validation and mutation of resources (default = 8443)| diff --git a/website/content/en/v1.0/troubleshooting.md b/website/content/en/v1.0/troubleshooting.md index 6819a794bd4c..1f0a5c0cfca2 100644 --- a/website/content/en/v1.0/troubleshooting.md +++ b/website/content/en/v1.0/troubleshooting.md @@ -248,6 +248,7 @@ To avoid this discrepancy between `maxPods` and the supported pod density of the 1. Enable [Prefix Delegation](https://www.eksworkshop.com/docs/networking/prefix/) to increase the number of allocatable IPs for the ENIs on each instance type 2. Reduce your `maxPods` value to be under the maximum pod density for the instance types assigned to your NodePods 3. Remove the `maxPods` value from your [`kubeletConfiguration`]({{}}) if you no longer need it and instead rely on the defaulted values from Karpenter and EKS AMIs. +4. Set [RESERVED_ENIS]({{}})=1 in your Karpenter configuration to account for the reserved ENI when using Security Groups for Pods. For more information on pod density, view the [Pod Density Section in the NodePools doc]({{}}).