Terraform Module: 🚀 Data & AI/ML Kubernetes Add-ons ⚙️

This Terraform module contains commonly used Data & AI/ML related Kubernetes add-ons that are typically included in Data on EKS blueprints. The purpose of this module is to provide users with the flexibility to select and customize the add-ons they require while leveraging the Data on EKS blueprints.

⚠️ Important Note

Users can consume this Terraform module in their projects to deploy any of the available addons. We will continue to maintain and update the existing Data/ML add-ons. However, we kindly request that you refrain from submitting Pull Requests (PRs) to add new addons at the moment, unless there is a supported blueprint available in the Data on EKS repository. The Apache and CNCF communities offer numerous open-source Data and ML add-ons, and while we appreciate their value, supporting all of them poses challenges.

Your understanding and cooperation are highly appreciated. 🙏

Usage

Create Addon with the following example using Terraform registry. Checkout the complete example under test folder.

module "eks_data_addons" {
  source = "aws-ia/eks-data-addons/aws"
  version = "~> 1.0" # ensure to update this to the latest/desired version

  oidc_provider_arn = module.eks.oidc_provider_arn

  # Example to deploy AWS Neuron Device Plugin for Trainium and Inferentia instances
  enable_aws_efa_k8s_device_plugin = true

  # Example to deploy EFA K8s Device Plugin for GPU/Neuron instances
  enable_aws_efa_k8s_device_plugin = true

  # Example to deploy NVIDIA GPU Operator
  enable_nvidia_gpu_operator = true

  # Example to deploy Spark Operator Helm Chart
  enable_spark_opertor = true

  # Example to deploy Flink Operator Helm Chart
  enable_flink_operator = true

  # Example to deploy Apache YuniKorn Helm Chart
  enable_yunikorn = true

  # Example that uses ECR authentication for a particular registry ID
  enable_emr_spark_operator = var.enable_emr_spark_operator
  emr_spark_operator_helm_config = {
    repository_username = data.aws_ecr_authorization_token.token.user_name
    repository_password = data.aws_ecr_authorization_token.token.password
  }

  # Example to deploy Helm chart that uses IAM Role for ServiceAccounts. You can disable `create_irsa` and bring your own IAM role.
  enable_spark_history_server = var.enable_emr_spark_operator
  spark_history_server_helm_config = {
    create_irsa = true
    values = [
      templatefile("${path.module}/test/helm-values/spark-history-server-values.yaml", {
        s3_bucket_name   = module.s3_bucket.s3_bucket_id
        s3_bucket_prefix = aws_s3_object.this.key
      })
    ]
  }
}

Support & Feedback

Important

EKS Blueprints for Terraform is maintained by AWS Solution Architects. It is not part of an AWS service and support is provided as a best-effort by the EKS Blueprints community. To provide feedback, please use the issues templates provided. If you are interested in contributing to EKS Blueprints, see the Contribution guide.

Requirements

Name	Version
terraform	>= 1.0.0
aws	>= 3.72
helm	>= 2.4.1

Providers

Name	Version
aws	>= 3.72
helm	>= 2.4.1

Modules

Name	Source	Version
spark_history_server_irsa	./irsa	n/a

Resources

Name	Type
helm_release.airflow	resource
helm_release.aws_efa_k8s_device_plugin	resource
helm_release.aws_neuron_device_plugin	resource
helm_release.cnpg_operator	resource
helm_release.dask_operator	resource
helm_release.daskhub	resource
helm_release.emr_flink_operator	resource
helm_release.emr_spark_operator	resource
helm_release.flink_operator	resource
helm_release.jupyterhub	resource
helm_release.karpenter_resources	resource
helm_release.kubecost	resource
helm_release.kuberay_operator	resource
helm_release.mlflow_tracking	resource
helm_release.nvidia_device_plugin	resource
helm_release.nvidia_gpu_operator	resource
helm_release.nvidia_triton_server	resource
helm_release.pinot	resource
helm_release.qdrant	resource
helm_release.spark_history_server	resource
helm_release.spark_operator	resource
helm_release.strimzi_kafka_operator	resource
helm_release.superset	resource
helm_release.trino	resource
helm_release.volcano	resource
helm_release.yunikorn	resource
aws_partition.current	data source
aws_region.current	data source

Inputs

Name	Description	Type	Default	Required
airflow_helm_config	Airflow Helm Chart config	`any`	`{}`	no
aws_efa_k8s_device_plugin_helm_config	EFA K8s Plugin add-on Helm Chart config	`any`	`{}`	no
aws_neuron_device_plugin_helm_config	AWS Neuron Device Plugin Helm Chart config	`any`	`{}`	no
cnpg_operator_helm_config	CloudNative PG Operator Helm Chart config	`any`	`{}`	no
dask_operator_helm_config	Dask Operator add-on configurations	`any`	`{}`	no
daskhub_helm_config	DaskHub add-on configurations	`any`	`{}`	no
emr_flink_operator_helm_config	Helm configuration for Flink Operator with EMR Runtime	`any`	`{}`	no
emr_spark_operator_helm_config	Helm configuration for Spark Operator with EMR Runtime	`any`	`{}`	no
enable_airflow	Enable Airflow add-on	`bool`	`false`	no
enable_aws_efa_k8s_device_plugin	Enable EFA K8s Plugin add-on	`bool`	`false`	no
enable_aws_neuron_device_plugin	Enable AWS Neuron Device Plugin add-on	`bool`	`false`	no
enable_cnpg_operator	Enable CloudNative PG Operator add-on	`bool`	`false`	no
enable_dask_operator	Enable Dask Operator add-on	`bool`	`false`	no
enable_daskhub	Enable DaskHub	`bool`	`false`	no
enable_emr_flink_operator	Enable the Flink Operator to run Flink application with EMR Runtime	`bool`	`false`	no
enable_emr_spark_operator	Enable the Spark Operator to submit jobs with EMR Runtime	`bool`	`false`	no
enable_flink_operator	Enable Flink Operator add-on	`bool`	`false`	no
enable_jupyterhub	Enable Jupyterhub Add-On	`bool`	`false`	no
enable_karpenter_resources	Enable Karpenter Resources (NodePool and EC2NodeClass)	`bool`	`false`	no
enable_kubecost	Enable Kubecost add-on	`bool`	`false`	no
enable_kuberay_operator	Enable Kuberay Operator add-on	`bool`	`false`	no
enable_mlflow_tracking	Enable MLflow Tracking add-on	`bool`	`false`	no
enable_nvidia_device_plugin	Enable NVIDIA Device Plugin add-on	`bool`	`false`	no
enable_nvidia_gpu_operator	Enable NVIDIA GPU Operator add-on	`bool`	`false`	no
enable_nvidia_triton_server	Enable NVIDIA Triton Server add-on	`bool`	`false`	no
enable_pinot	Enable Apache Pinot Add-On	`bool`	`false`	no
enable_qdrant	Enable Qdrant Vector Database add-on	`bool`	`false`	no
enable_spark_history_server	Enable Spark History Server add-on	`bool`	`false`	no
enable_spark_operator	Enable Spark on K8s Operator add-on	`bool`	`false`	no
enable_strimzi_kafka_operator	Enable the Strimzi Kafka Operator	`bool`	`false`	no
enable_superset	Enable Apache Supeset add-on	`bool`	`false`	no
enable_trino	Enable Trino add-on	`bool`	`false`	no
enable_volcano	Enable volcano scheduler add-on	`bool`	`false`	no
enable_yunikorn	Enable Apache YuniKorn K8s scheduler add-on	`bool`	`false`	no
flink_operator_helm_config	Flink Operator Helm Chart config	`any`	`{}`	no
jupyterhub_helm_config	Helm configuration for JupyterHub	`any`	`{}`	no
karpenter_resources_helm_config	Karpenter Resources Helm Chart config	`any`	`{}`	no
kubecost_helm_config	Kubecost Helm Chart config	`any`	`{}`	no
kuberay_operator_helm_config	Helm configuration for Kuberay Operator	`any`	`{}`	no
mlflow_tracking_helm_config	MLflow Tracking add-on Helm Chart config	`any`	`{}`	no
nvidia_device_plugin_helm_config	NVIDIA Device Plugin Helm Chart config	`any`	`{}`	no
nvidia_gpu_operator_helm_config	Helm configuration for NVIDIA GPU Operator	`any`	`{}`	no
nvidia_triton_server_helm_config	Helm configuration for NVIDIA Triton Server	`any`	`{}`	no
oidc_provider_arn	The ARN of the cluster OIDC Provider	`string`	n/a	yes
pinot_helm_config	Apache Pinot Helm Chart config	`any`	`{}`	no
qdrant_helm_config	Helm configuration for Qdrant	`any`	`{}`	no
spark_history_server_helm_config	Helm configuration for Spark History Server	`any`	`{}`	no
spark_operator_helm_config	Helm configuration for Spark K8s Operator	`any`	`{}`	no
strimzi_kafka_operator_helm_config	Helm configuration for Strimzi Kafka Operator	`any`	`{}`	no
superset_helm_config	Apache Supeset Helm Chart config	`any`	`{}`	no
trino_helm_config	Trino Helm Chart config	`any`	`{}`	no
volcano_helm_config	Volcano scheduler add-on configurations	`any`	`{}`	no
yunikorn_helm_config	Helm configuration for Apache YuniKorn	`any`	`{}`	no

Outputs

Name	Description
airflow	Airflow Helm Chart metadata
aws_efa_k8s_device_plugin	AWS EFA K8s Plugin Helm Chart metadata
aws_neuron_device_plugin	AWS Neuron Device Plugin Helm Chart metadata
dask_hub	Dask Hub Helm Chart metadata
dask_operator	Dask Operator Helm Chart metadata
emr_spark_operator	EMR Spark Operator Helm Chart metadata
flink_operator	Flink Operator Helm Chart metadata
jupyterhub	Jupyterhub Helm Chart metadata
kubecost	Kubecost Helm Chart metadata
kuberay_operator	Kuberay Operator Helm Chart metadata
nvidia_gpu_operator	Nvidia GPU Operator Helm Chart metadata
pinot	Apache Pinot Helm Chart metadata
spark_history_server	Spark History Server Helm Chart metadata
spark_operator	Spark Operator Helm Chart metadata
strimzi_kafka_operator	Strimzi Kafka Operator Helm Chart metadata
volcano	Volcano Batch Scheduler Helm Chart metadata
yunikorn	Yunikorn Helm Chart metadata

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Usage

Support & Feedback

Requirements

Providers

Modules

Resources

Inputs

Outputs

Files

README.md

Latest commit

History

README.md

File metadata and controls

Usage

Support & Feedback

Requirements

Providers

Modules

Resources

Inputs

Outputs