Skip to content

Commit

Permalink
docs: AL2023 AMI Family docs
Browse files Browse the repository at this point in the history
  • Loading branch information
@jmdeal
jmdeal committed Feb 23, 2024
1 parent ce14661 commit 4ff5072
Showing 1 changed file with 229 additions and 4 deletions.
233 changes: 229 additions & 4 deletions website/content/en/preview/concepts/nodeclasses.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ Refer to the [NodePool docs]({{<ref "./nodepools" >}}) for settings applicable t

## spec.amiFamily

AMIFamily is a required field, dictating both the default bootstrapping logic for nodes provisioned through this `EC2NodeClass` but also selecting a group of recommended, latest AMIs by default. Currently, Karpenter supports `amiFamily` values `AL2`, `Bottlerocket`, `Ubuntu`, `Windows2019`, `Windows2022` and `Custom`. GPUs are only supported by default with `AL2` and `Bottlerocket`. The `AL2` amiFamily does not support ARM64 GPU instance types unless you specify custom [`amiSelectorTerms`]({{<ref "#specamiselectorterms" >}}). Default bootstrapping logic is shown below for each of the supported families.
AMIFamily is a required field, dictating both the default bootstrapping logic for nodes provisioned through this `EC2NodeClass` but also selecting a group of recommended, latest AMIs by default. Currently, Karpenter supports `amiFamily` values `AL2`, `AL2023`, `Bottlerocket`, `Ubuntu`, `Windows2019`, `Windows2022` and `Custom`. GPUs are only supported by default with `AL2` and `Bottlerocket`. The `AL2` amiFamily does not support ARM64 GPU instance types unless you specify custom [`amiSelectorTerms`]({{<ref "#specamiselectorterms" >}}). Default bootstrapping logic is shown below for each of the supported families.

### AL2

Expand All @@ -184,6 +184,34 @@ exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
--//--
```

### AL2023

```yaml
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: application/node.eks.aws
# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
name: test-cluster
apiServerEndpoint: https://example.com
certificateAuthority: ca-bundle
cidr: 10.100.0.0/16
kubelet:
config:
maxPods: 110
flags:
- --node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test
--//--
```

### Bottlerocket

```toml
Expand Down Expand Up @@ -581,6 +609,17 @@ spec:
encrypted: true
```

### AL2023
```yaml
spec:
blockDeviceMappings:
- deviceName: /dev/xvda
ebs:
volumeSize: 20Gi
volumeType: gp3
encrypted: true
```

### Bottlerocket
```yaml
spec:
Expand Down Expand Up @@ -646,6 +685,9 @@ The disks must be formatted & mounted in a RAID0 and be the underlying filesyste

On AL2, Karpenter automatically configures the disks through an additional boostrap argument (`--local-disks raid0`). The device name is `/dev/md/0` and its mount point is `/mnt/k8s-disks/0`. You should ensure any additional disk setup does not interfere with these.

#### AL2023
On AL2023, local disks are always configured as raid0. Like AL2, the device name is `/dev/md/0` and its mount point is `/mnt/k8s-disks/0`. You should ensure any additional disk setup does not interfere with these.

#### Others

For all other AMI families, you must configure the disks yourself. Check out the [`setup-local-disks`](https://github.com/awslabs/amazon-eks-ami/blob/master/files/bin/setup-local-disks) script in [amazon-eks-ami](https://github.com/awslabs/amazon-eks-ami) to see how this is done for AL2.
Expand Down Expand Up @@ -717,7 +759,30 @@ Consider the following example to understand how your custom UserData will be me

```bash
#!/bin/bash
echo "Running custom user data script"
echo "Running custom user data script (bash)"
```

#### Merged UserData (bash)

```bash
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script (bash)"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash -xe
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
/etc/eks/bootstrap.sh 'test-cluster' --apiserver-endpoint 'https://test-cluster' --b64-cluster-ca 'ca-bundle' \
--use-max-pods false \
--kubelet-extra-args '--node-labels=karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=test --max-pods=110'
--//--
```

#### Passed-in UserData (MIME)
Expand All @@ -730,7 +795,7 @@ Content-Type: multipart/mixed; boundary="BOUNDARY"
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script"
echo "Running custom user data script (mime)"
--BOUNDARY--
```
Expand All @@ -745,7 +810,7 @@ Content-Type: multipart/mixed; boundary="//"
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Running custom user data script"
echo "Running custom user data script (mime)"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
Expand Down Expand Up @@ -775,6 +840,166 @@ spec:
```
{{% /alert %}}

### AL2023

* Your UserData may be in one of three formats: a [MIME multi part archive](https://cloudinit.readthedocs.io/en/latest/topics/format.html#mime-multi-part-archive), a NodeConfig YAML / JSON string, or a shell script.
* Karpenter will transform your custom UserData into a MIME part, if necessary, and then create a MIME multi-part archive. This archive will consist of a generated NodeConfig, containing Karpenter's default values, followed by the transformed custom UserData. For more information on how this UserData is handled by the AMI, refer to the [AL2023 EKS Optimized AMI docs](https://awslabs.github.io/amazon-eks-ami/nodeadm/doc/examples/).

#### Passed-in UserData (NodeConfig)

```yaml
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
maxPods: 42
```

#### Merged UserData (NodeConfig)

```shell
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: application/node.eks.aws
# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
apiServerEndpoint: https://test-cluster
certificateAuthority: cluster-ca
cidr: 10.100.0.0/16
name: test-cluster
kubelet:
config:
clusterDNS:
- 10.100.0.10
maxPods: 118
flags:
- --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"
--//
Content-Type: application/node.eks.aws
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
maxPods: 42
--//--
```

#### Passed-in UserData (bash)

```shell
#!/bin/bash
echo "Hello, AL2023!"
```

#### Merged UserData (bash)

```shell
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"
--//
Content-Type: application/node.eks.aws
# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
apiServerEndpoint: https://test-cluster
certificateAuthority: cluster-ca
cidr: 10.100.0.0/16
name: test-cluster
kubelet:
config:
clusterDNS:
- 10.100.0.10
maxPods: 118
flags:
- --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"
--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
echo "Hello, AL2023!"
--//--
#### Passed-in UserData (MIME)
```shell
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"
--//
```

#### Merged UserData (MIME)

```shell
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: application/node.eks.aws

# Karpenter Generated NodeConfig
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
cluster:
apiServerEndpoint: https://test-cluster
certificateAuthority: cluster-ca
cidr: 10.100.0.0/16
name: test-cluster
kubelet:
config:
clusterDNS:
- 10.100.0.10
maxPods: 118
flags:
- --node-labels="karpenter.sh/capacity-type=on-demand,karpenter.sh/nodepool=default"

--//
Content-Type: application/node.eks.aws

apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
maxPods: 42
--//
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Hello, AL2023!"
--//--
```

### Bottlerocket

* Your UserData must be valid TOML.
Expand Down

0 comments on commit 4ff5072

Please sign in to comment.