Workload API

• 1: Pod Group Disruption and Priority
• 2: PodGroup Scheduling Policies
• 3: Topology-Aware Workload Scheduling

apiVersion: scheduling.k8s.io/v1alpha2
kind: Workload
metadata:
  name: training-job-workload
  namespace: some-ns
spec:
  controllerRef:
    apiGroup: batch
    kind: Job
    name: training-job
  podGroupTemplates:
  - name: workers
    schedulingPolicy:
      gang:
        # The gang is schedulable only if 4 pods can run at once
        minCount: 4
    priorityClassName: high-priority # Only applicable with WorkloadAwarePreemption feature gate
    disruptionMode: PodGroup # Only applicable with WorkloadAwarePreemption feature gate

1 - Pod Group Disruption and Priority

PodGroup can declare a disruption mode. This mode dictates how the scheduler can disrupt a running PodGroup, for example to accommodate a higher priority PodGroup. A PodGroup also has a priority, which overrides the priority of the individual pods from the group for workload-aware preemption events.

Disruption mode types

The API supports two disruption modes: Pod and PodGroup. The default one is Pod.

Pod

The Pod mode instructs the scheduler to treat all Pods in the group as separate entities, allowing independent disruption of a single pod from a PodGroup.

PodGroup

The PodGroup mode emphasizes "all-or-nothing" semantics for disruption. It instructs the scheduler that all pods from the PodGroup have to be disrupted together.

Pod group priority

PodGroup uses the same concept of PriorityClass as single Pods. Once you have created one or more PriorityClasses, you can create a PodGroup that specifies one of those PriorityClass names in its specification. The priority admission controller uses the priorityClassName field and populates the integer value of the priority. If the priority class is not found, the PodGroup is rejected. When priorityClassName is not set for a PodGroup, Kubernetes looks for a default (a PriorityClass with globalDefault set true) If there is no PriorityClass with globalDefault set true, a PodGroup with no priorityClassName has priority zero.

The priority of the PodGroup is an authorative priority for all pods in the group during workload-aware preemption events, even when priorities of individual pods forming this PodGroup differ.

The following YAML is an example of a PodGroup configuration that uses the high-priority PriorityClass, which maps to the integer priority value of 1000000. The priority admission controller checks the specification and resolves the priority of the PodGroup to 1000000.

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  namespace: ns-1
  name: job-1
spec:
  priorityClassName: high-priority

What's next

• Read about Workload-Aware Preemption algorithm.
• Learn about the Workload API.

2 - PodGroup Scheduling Policies

Every PodGroup must declare a scheduling policy in its spec.schedulingPolicy field. This policy dictates how the scheduler treats the collection of Pods in the group.

Policy types

The schedulingPolicy field supports two policy types: basic and gang. You must specify exactly one.

Basic policy

The basic policy instructs the scheduler to evaluate all Pods on a best-effort basis. Unlike the gang policy, a PodGroup using the basic policy is considered feasible regardless of how many of its Pods are currently schedulable.

The primary reason to use the basic policy is to organize Pods into a group for better observability and management, while still evaluating them together within a single, atomic PodGroup scheduling cycle.

This policy is suited for groups that do not require simultaneous startup but logically belong together, or to open the way for group-level constraints that do not imply "all-or-nothing" placement.

schedulingPolicy:
  basic: {}

Gang policy

The gang policy enforces "all-or-nothing" scheduling. This is essential for tightly-coupled workloads where partial startup results in deadlocks or wasted resources.

This can be used for Jobs or any other batch process where all workers must run concurrently to make progress.

The gang policy requires a minCount field, which is the minimum number of Pods that must be schedulable simultaneously for the group to be feasible:

schedulingPolicy:
  gang:
    # The number of Pods that must be schedulable simultaneously
    # for the group to be admitted.
    minCount: 4

Setting policies via PodGroupTemplates

When using the Workload API, you define scheduling policies inside PodGroupTemplates. The workload controller copies the policy from the template into each PodGroup it creates, making the PodGroup self-contained. Changes to the Workload's templates only affect newly created PodGroups, not existing ones.

For standalone PodGroups (created without a Workload), you set spec.schedulingPolicy directly on the PodGroup itself.

What's next

• See the PodGroup API for how policies are carried at runtime.
• Learn about the Workload API that defines PodGroupTemplates.
• Read about PodGroup scheduling.
• Read about the gang scheduling algorithm.

3 - Topology-Aware Workload Scheduling

Topology-Aware Scheduling (TAS) is a feature of the Workload API that optimizes the placement of pods within the cluster.

TAS ensures that all pods within a PodGroup are co-located into a specific topology domain, such as a single server rack or zone. This minimizes inter-pod communication latency and prevents workload fragmentation across the cluster infrastructure.

Topology-aware scheduling with gang scheduling policy

When applied to PodGroups with gang scheduling policy, TAS simulates the potential assignment (placement) of the full group of pods at once. It guarantees that at least the specified minCount pods can fit together into the same topology domain before committing resources. If no feasible placement is found, the entire PodGroup becomes unschedulable.

This is the recommended approach for workloads like distributed AI and ML training that strictly require proximity to minimize inter-pod communication latency.

If new pods are added to the PodGroup where some pods are already scheduled (for example, if pods are recreated), the scheduler will force all new incoming pods to land on the exact same topology domain where the existing pods currently reside. If that specific domain lacks sufficient capacity for the new pods, the pods will remain pending - even if it means that less than minCount pods are scheduled at this point.

Topology-aware scheduling with basic scheduling policy

Using TAS with basic scheduling policy may exhibit inconsistent behavior. The scheduler may only observe a subset of pods when entering the PodGroup scheduling cycle - therefore placement feasibility is only evaluated for the observed pods, rather than the entire PodGroup. To partially mitigate this limitation, you can use scheduling gates to hold off PodGroup scheduling until all pods within the PodGroup are in the scheduling queue.

If no feasible placement is found for the entire PodGroup, only a subset of pods may be scheduled, and they are guaranteed to meet the scheduling constraints.

If new pods are added to the PodGroup where some pods are already scheduled, the scheduler will act the same as in case of gang policy - forcing the new pods into the same domain, unless there is insufficient capacity (in which case the new pods will remain pending).

API configuration: scheduling constraints

Every PodGroup (or PodGroupTemplate) may optionally declare the schedulingConstraints field, which is interpreted by the placement-based PodGroup scheduling algorithm. If constraints are defined in PodGroupTemplate, they will be copied to referencing PodGroups.

As of Kubernetes v1.36, the API supports topology constraints.

Topology constraint

To define a topology constraint for a PodGroup you need to set a key, which corresponds to a Kubernetes node label, representing the target topology domain (for example, a rack or a zone). The scheduler strictly enforces that all pods within the PodGroup are placed onto nodes that share the exact same value for this specified label.

Here is an example of a PodGroup configured with a topology constraint:

apiVersion: scheduling.k8s.io/v1alpha2
kind: PodGroup
metadata:
  name: example-podgroup
spec:
  schedulingPolicy:
    gang:
      minCount: 4
  schedulingConstraints:
    topology:
      - key: topology.example.com/rack

What's next

• Learn about pod group policies.
• Learn about plugins related Topology-aware Scheduling
• Read about gang scheduling algorithm.