From 34cd93074d08f16d05a6f7c79ca90e5a7edaf0fb Mon Sep 17 00:00:00 2001 From: "Hsien-Cheng(Ryan) Huang" Date: Mon, 9 Sep 2024 16:17:00 -0500 Subject: [PATCH] [YUNIKORN-2814] Add YuniKorn on Spark(AWS) link to Run spark jobs page (#467) Closes: #467 Signed-off-by: Craig Condit --- docs/user_guide/workloads/run_spark.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/user_guide/workloads/run_spark.md b/docs/user_guide/workloads/run_spark.md index e5e13736e23..83a26610d16 100644 --- a/docs/user_guide/workloads/run_spark.md +++ b/docs/user_guide/workloads/run_spark.md @@ -164,3 +164,10 @@ to YuniKorn and being considered as a job. The job is scheduled and running as t YuniKorn allocates the driver pod to a node, binds the pod and starts all the containers. Once the driver pod gets started, it requests for a bunch of executor pods to run its tasks. Those pods will be created in the same namespace as well and scheduled by YuniKorn as well. + +## Using YuniKorn as a custom scheduler for Apache Spark on Amazon EMR on EKS + +YuniKorn can be configured as a custom scheduler for Apache Spark jobs on Amazon EMR on EKS. This setup allows our resource management and scheduling algorithms on Kubernetes clusters. + +For a detailed guide on how to set up YuniKorn with Apache Spark on Amazon EMR on EKS, please refer to the [AWS EMR documentation](https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/tutorial-yunikorn.html). +