Skip to content

Latest commit

 

History

History
102 lines (80 loc) · 4.82 KB

02-deploying-eks-cluster.md

File metadata and controls

102 lines (80 loc) · 4.82 KB
Raw

Deploying an EKS cluster

Before proceeding, it's recommended to check your organization's preferred networking and security settings for EKS clusters. Flyte doesn't require anything beyond what the Kubernetes nodes require to operate::

  • Connection to the EKS control plane (also called API Server)

In this tutorial we make use of the following configuration:

  • API server endpoint access: Public and Private
  • Node group using public subnets
  • VPC already configured with both Public and Private subnets
  • Public subnets enabled to Auto-assign public IPV4 address

Learn more about EKS networking

  1. Make sure that your system has the following components installed:
  1. If you haven't done it previously, complete the awscli quick configuration
  2. Go to the AWS Management console > Virtual Private Cloud > Subnets and take note of both the Public and Private subnet IDs available in the supported Availability Zones on the VPC you plan to use to deploy Flyte:

  1. Go to the terminal and submit the command to create the EKS control plane, following these recommendations:
  • Pick a Kubernetes version >= 1.19
  • Give the cluster an informative name (eg. flyte-eks-cluster)
eksctl create cluster --name my-cluster --region region-code  --version 1.25 --vpc-private-subnets private-subnet-ID1,private-subnet-ID2 --vpc-public-subnets public-subnetID1,public-subnetID2 --without-nodegroup

After a few minutes, the deployment should finish with an output similar to:

EKS cluster "my-cluster" in "region-code" region is ready
  1. From the AWS Management Console go to EKS
  2. Click on your cluster name and select the Networking tab
  3. Click on Manage networking
  4. Select Public and private as the Cluster endpoint access mode and Save changes

Create an EKS node group

In this step we'll deploy the Kubernetes worker nodes where the actual workloads will run.

  1. From the AWS Management console, go to EKS
  2. Select the cluster you created in the previous section
  3. Go to Compute and click on Add node group
  4. Assign a name to the node group
  5. In the Node IAM role menu, select the <eks-node-role> created in the Roles and Service accounts section
  6. For learning purposes, do not use launch templates, labels or taints
  7. Choose the default Amazon Linux 2 (AL2_x86_x64) AMI type
  8. Use On-demand Capacity type
  9. Instance type and size can be chosen based on your devops requirements.

NOTE: Previous testing suggests that t3.xlarge provides the required resources to run the example workflows referenced in this tutorial.

  1. Keep all other options by default.
  2. Review the configuration and click Create

Learn more about AWS managed node groups

Connecting to the EKS cluster

Now, you have to “let your terminal know” where’s the Kubernetes cluster you’re trying to connect to.

Use your AWS account access keys to run the following commands, updating the kubectl config and switching to the new EKS cluster context:

  1. Store your AWS login information in environment variables so they can be grabbed by awscli:
export AWS_ACCESS_KEY_ID=<YOUR-AWS-ACCOUNT-ACCESS-KEY-ID>
export AWS_SECRET_ACCESS_KEY=<YOUR-AWS-SECRET-ACCESS-KEY>
export AWS_SESSION_TOKEN=<YOUR-AWS-SESSION-TOKEN>
  1. Switch to the new EKS cluster’s context:
aws eks update-kubeconfig --name <my-cluster> --region <region>
  1. Verify the context has switched:
$ kubectl config current-context
arn:aws:eks:<region>:<AWS_ACCOUNT_ID>:cluster/<Name-EKS-Cluster>
  1. Verify there are no Pod resources created (this step also validates connectivity to the EKS API Server)
$ kubectl get pods
No resources found in default namespace.

Create an S3 bucket

This is the blob storage resource that will be used by FlytePropeller and DataCatalog to store metadata for versioning and other use cases.

  1. From the AWS Management Console, go to S3
  2. Create a bucket leaving Block all public access enabled
  3. Choose the same region and VPC as your EKS cluster
  4. Take note of the name as it will be used in your Helm values file

Next: Configure roles and service accounts