Do non-preemptible pods on Kueue still preempt lower-priority pods?

[jenniferlai43]

Hi folks, had a question about preemption + Kueue to better my understanding.

My understanding of workload priority class is that if I set pod-level priority, it is used for both scheduling and pod-priority based on this kueue documentation.

In k8s, a non-preempting priority class means that preemptionPolicy: Never with higher priority can be scheduled ahead of other pods, but not preempt existing pods. (k8s documentation)

In a simple setup, I set 2 jobs to have preemptionPolicy: Never - job A has PriorityClass with value 100, job B has PriorityClass will value 200.

Simple example is below:

PriorityClass:

# pod-priority.yaml
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: pod-nonpreemptible-mid-priority
value: 100
globalDefault: false
description: "This is a non-preemptible mid priority class. It cannot preempt other pods."
preemptionPolicy: Never
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: pod-nonpreemptible-high-priority
value: 200
globalDefault: false
description: "This is a non-preemptible high priority class. It cannot preempt other pods."
preemptionPolicy: Never
---

Jobs:

# mid-prio-batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: mid-prio-batch-job
  annotations:
    kueue.x-k8s.io/queue-name: test-queue
spec:
  completions: 4
  parallelism: 2
  completionMode: Indexed
  backoffLimit: 0
  template:
    metadata:
      labels:
        job-name: mid-prio-batch-job
    spec:
      priorityClassName: pod-nonpreemptible-mid-priority
      containers:
        - name: worker
          image: busybox
          command: [ "sh", "-c" ]
          args:
            - |
              echo 'Job $(JOB_COMPLETION_INDEX) starting';
              sleep 600;
              echo 'Job $(JOB_COMPLETION_INDEX) completed'
          resources:
            requests:
              cpu: "1"
              memory: "1Gi"
      restartPolicy: Never

# high-prio-batch-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: high-prio-batch-job
  annotations:
    kueue.x-k8s.io/queue-name: test-queue
spec:
  completions: 4
  parallelism: 2
  completionMode: Indexed
  backoffLimit: 0
  template:
    metadata:
      labels:
        job-name: high-prio-batch-job
    spec:
      priorityClassName: pod-nonpreemptible-high-priority
      containers:
        - name: worker
          image: busybox
          command: [ "sh", "-c" ]
          args:
            - |
              echo 'Job $(JOB_COMPLETION_INDEX) starting';
              sleep 600;
              echo 'Job $(JOB_COMPLETION_INDEX) completed'
          resources:
            requests:
              cpu: "1"
              memory: "1Gi"
      restartPolicy: Never

ClusterQueue:

# cluster-queue.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  name: preemptible-cluster-queue
spec:
  namespaceSelector: {} # match all.
  flavorFungibility:
    whenCanBorrow: Borrow
    whenCanPreempt: TryNextFlavor
  preemption:
    withinClusterQueue: LowerPriority
  resourceGroups:
  - coveredResources: ["cpu", "memory", "pods"]
    flavors:
    - name: default
      resources:
      - name: "cpu"
        nominalQuota: 2
      - name: "memory"
        nominalQuota: 2Gi
      - name: "pods"
        nominalQuota: 2

LocalQueue:

# local-queue.yaml
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: test-queue
spec:
  clusterQueue: preemptible-cluster-queue

When I let the mid-prio-batch-job run, then before it completes, start the high-prio-batch-job, I can see the mid-prio job is suspended and pods are deleted, then the high-prio job starts.

For my understanding, is pod-level preemptionPolicy from PriorityClass not taken into account for workload preemption? If it was, I would assume that mid-prio-batch-job could not be preempted, because the high-prio-batch-job has specified preemptionPolicy: Never. Let me know if I am mistaken!

[mimowo]

Yes, we currently don't consider preemptionPolicy (only the priorities) for workload-level scheduling and preemption (here are the look ups, but int32 is returned.

There are a couple of related topics which can help for your use-case (let us know if this helps):

• workload-priority class, allowing to de-couple the workload-level preemption from pod priorities
• CQ-level preemption allows to to disable preemption for a subset of CQs.

Now, I think it might be considered a bug (or omission) that we don't consider the preemptionPolicy: Never, but I'm not sure about the reasoning process. Maybe @tenzen-y remembers / knows the details. When deciding if this is a bug or not we need to be careful about existing installations.

[jenniferlai43]

Thanks for the response! Not a a blocker for me, I've been using pod-level priority (just the value) in tandem with CQ-level preemption to get around this - I was just curious how pod-level preemption settings play with workload preemption.

[mimowo]

I think it is ok that Kueue does not respsect preemptionPolicy for pods, because we don't preempt pods directly, just workloads.

However, I think the fact that preemptionPolicy is not respected for workloads priority class sounds like a bug, TBH. Feel free to open an issue.