document using AppWrappers to run wrapped custom Jobs (#3978)

* document using AppWrappers to run wrapped custom Jobs

* address review comments

site/content/en/docs/tasks/dev/integrate_a_custom_job.md

@@ -9,9 +9,11 @@ description: >
 Kueue has built-in integrations for several Job types, including
 Kubernetes batch Job, MPIJob, RayJob and JobSet.
 
-There are two options for adding an additional integration for a Job-like CRD with Kueue:
-- As part of the Kueue repository
-- Writing an external controller
+There are three options for using Kueue to manage Job-like CRDs that lack built-in integrations.
+- Leverage the built-in AppWrapper integration by wrapping instances of the custom Job in an AppWrapper.
+  See [Running a Wrapped Custom Workload](/docs/tasks/run/wrapped_custom_workload) for details.
+- Build a new integration as part of the Kueue repository.
+- Build a new integration as an external controller.
 
 This guide is for [platform developers](/docs/tasks#platform-developer) and describes how
 to build a new integration. Integrations should be built using the APIs provided by

site/content/en/docs/tasks/run/wrapped_custom_workload.md

@@ -0,0 +1,58 @@
+---
+title: "Run A Wrapped Custom Workload"
+linkTitle: "Custom Workload"
+date: 2025-01-14
+weight: 7
+description: >
+  Use an AppWrapper to Run a Custom Workload on Kueue.
+---
+
+This page shows how to use [AppWrappers](https://project-codeflare.github.io/appwrapper/) to make
+Kueue's scheduling and resource management capabilities available to Workload types that do not have a dedicated
+Kueue integration.  For Workloads that use `PodSpecTemplates` in their definition, this can provide
+a significantly easier approach than [building a custom integration](/docs/tasks/dev/integrate_a_custom_job)
+to enable the use of Kueue with a custom Workload type.
+
+This guide is for [batch users](/docs/tasks#batch-user) that have a basic understanding of Kueue. For more information, see [Kueue's overview](/docs/overview).
+
+## Before you begin
+
+1. Make sure you are using Kueue v0.11.0 version or newer and AppWrapper v1.0.0 or newer.
+
+2. Follow the steps in [Run AppWrappers](/docs/tasks/run/appwrappers/#before-you-begin)
+to learn how to enable and configure the `codeflare.dev/appwrapper` integration.
+
+## Example using LeaderWorkerSets as the Custom Workload
+
+We use [LeaderWorkerSets](https://github.com/kubernetes-sigs/lws) to explain how to
+run a workload of a custom type inside an AppWrapper.
+
+1. Follow the [install](https://github.com/kubernetes-sigs/lws/blob/main/docs/setup/install.md)
+instructions for LeaderWorkerSets.
+
+2. Edit the `appwrapper-manager-role` `ClusterRole` to add the stanza below to allow
+the appwrapper controller to manipulate LeaderWorketSets.
+```yaml
+- apiGroups:
+  - leaderworkerset.x-k8s.io
+  resources:
+  - leaderworkersets
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - patch
+  - update
+  - watch
+```
+
+3. The AppWrapper containing the LeaderWorkerSet is shown below.
+In particular, notice how the `replicas` and `path` of each element of the `podSets` array
+corresponds to a `PodSpecTemplate` and replica count within `template`.
+This gives the AppWrapper controller enough information to enable
+it to "understand" the wrapped resource and provide Kueue the information it needs to
+manage it.
+
+{{< include "examples/jobs/appwrapper-leaderworkerset-sample.yaml" "yaml" >}}
+

site/static/examples/jobs/appwrapper-leaderworkerset-sample.yaml

@@ -0,0 +1,44 @@
+apiVersion: workload.codeflare.dev/v1beta2
+kind: AppWrapper
+metadata:
+  name: sample-lws
+  labels:
+    kueue.x-k8s.io/queue-name: user-queue
+spec:
+  components:
+  - podSets:
+    - path: "template.spec.leaderWorkerTemplate.leaderTemplate"
+      replicas: 2
+    - path: "template.spec.leaderWorkerTemplate.workerTemplate"
+      replicas: 3
+    template:
+      apiVersion: leaderworkerset.x-k8s.io/v1
+      kind: LeaderWorkerSet
+      metadata:
+        name: nginx-leaderworkerset
+        labels:
+          app: nginx
+      spec:
+        replicas: 2
+        leaderWorkerTemplate:
+          leaderTemplate:
+            spec:
+              containers:
+              - name: nginx-leader
+                image: registry.k8s.io/nginx-slim:0.27
+                resources:
+                  requests:
+                    cpu: "100m"
+                ports:
+                - containerPort: 80
+          size: 3
+          workerTemplate:
+            spec:
+              containers:
+              - name: nginx-worker
+                image: nginx:1.14.2
+                resources:
+                  requests:
+                    cpu: "200m"
+                ports:
+                - containerPort: 80