Skip to content

Commit

Permalink
[YUNIKORN-2919] Document spark-operator: Account spark.memory.offHeap…
Browse files Browse the repository at this point in the history
….size (#508)
  • Loading branch information
ryankert01 authored Dec 24, 2024
1 parent 56ed96b commit c496925
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions docs/user_guide/workloads/run_spark.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ under the License.
:::note
Pre-requisites:
- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn`
- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling
- Use spark-operator version >= 2.1.0 to enable support for YuniKorn gang scheduling
:::

### Install YuniKorn
Expand All @@ -48,7 +48,7 @@ helm install yunikorn yunikorn/yunikorn --create-namespace --namespace yunikorn
We should install `spark-operator` with `controller.batchScheduler.enable=true` and set `controller.batchScheduler.default=yunikorn` to enable Gang Scheduling. It's optional to set the default scheduler to YuniKorn since you can specify it later on, but it's recommended to do so.
Also, note that the total requested memory for the Spark job is the sum of memory requested for the driver and that for all executors, where each is computed as below:
* Driver requested memory = `spark.driver.memory` + `spark.driver.memoryOverhead`
* Executor requested memory = `spark.executor.memory` + `spark.executor.memoryOverhead` + `spark.executor.pyspark.memory`
* Executor requested memory = `spark.executor.memory` + `spark.executor.memoryOverhead` + `spark.executor.pyspark.memory` + `spark.memory.offHeap.size`

```shell script
helm repo add spark-operator https://kubeflow.github.io/spark-operator
Expand Down

0 comments on commit c496925

Please sign in to comment.