Kubernetes v1.37 Takes Workload-Aware Scheduling to Beta: Bin-Packing Batch Jobs on Bare Metal Without a Managed Autoscaler

The most expensive scheduling bug in Kubernetes never pages anyone. A distributed training job needs four pods to make progress; the scheduler places two, the other two sit pending, and the two running pods burn GPU-hours waiting for rendezvous peers that will never arrive. On a cloud cluster, the cluster autoscaler eventually papers over this by adding a node. On a bare-metal fleet with a fixed node count — the Hetzner machines a self-hosted PaaS already paid for — there is no new node coming. That half-scheduled gang just wedges capacity you own until somebody notices.

Kubernetes has been fixing this in the open for three releases under the name workload-aware scheduling, and v1.37 is the milestone that matters for small operators: the Workload and PodGroup APIs plus native gang scheduling graduated to Beta in the September 8 upstream announcement, one step from GA. Here is the timeline that got us here, pinned to releases so you know exactly what "goes Beta" means:

If you run a Cluster API fleet on owned hardware, this is the first Kubernetes release where scheduling whole workloads instead of individual pods is a supported-enough bet to plan around. The rest of this post works through what changes on a fixed node count, with numbers, and the exact checklist to opt in.

How the scheduler learned to see whole workloads​

The classic kube-scheduler sees one pod at a time. It pops a pod off the queue, filters nodes, scores them, binds — then repeats for the next pod, blind to the fact that the eight pods it just placed are one MPI job that deadlocks unless all eight land. Volcano and Kueue solved this years ago outside the tree with their own PodGroup CRDs and queueing layers, at the cost of running a second scheduler or a parallel admission system next to kube-scheduler.

The in-tree answer, tracked as KEP-4671, introduces two objects. A Workload is a static template describing a group's scheduling policy — today mostly gang with a minCount. A PodGroup is the runtime instance stamped out from that template, carrying live status conditions that mirror the member pods. Pods link to their group through the schedulingGroup.podGroupName field. The v1.36 refactor that split template from runtime exists for a performance reason worth knowing: per-replica sharding of status updates means the scheduler reads one PodGroup instead of watching and parsing whole Workload objects.

The scheduling behavior is what matters operationally. Member pods are held in PreEnqueue until the group's minCount is satisfiable; then the dedicated PodGroup scheduling cycle takes a single snapshot of cluster state, evaluates placements for the whole group against it, and binds atomically — all members move to binding together, or none do and the group backs off and retries. Already-bound pods are never unassigned by a later cycle, so a gang that landed stays landed even if a subsequent scale-up of the group cannot be satisfied immediately.

A minimal group looks like this:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: training-job-workload
spec:
  podGroupTemplates:
    - name: workers
      schedulingPolicy:
        gang:
          minCount: 4

and the runtime instance stamped from it:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-job-workers-pg
spec:
  workloadRef:
    workloadName: training-job-workload
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 4

In practice you will not hand-write these; workload controllers (Job, JobSet, LeaderWorkerSet) stamp out the PodGroups. But knowing the two-level shape matters the moment you debug why a group is stuck pending.

The bare-metal payoff, worked through with numbers​

Take a concrete fixed pool: four identical bare-metal nodes, 16 CPU each, running a mix of tenant pods and one nightly 4-pod batch gang where each pod requests 4 CPU. Total gang footprint is 16 CPU — exactly one node's worth — but the scheduler places pods wherever they fit.

Before: pod-at-a-time scheduling. Evening load occupies 46 of 64 CPUs, leaving 18 free but fragmented: 7 on node A, 7 on node B, 2 on node C, 2 on node D. The scheduler places two gang pods (A, B), and the other two go pending — no remaining node fits 4 CPU. The two running pods cannot start work; they hold 8 CPUs doing rendezvous retries.

Meanwhile two tenant worker pods requesting 4 CPU each also pend, because the largest remaining fragment is 3 CPU. The pool reads 84% allocated, but nearly a sixth of that allocation is gang pods making zero progress, and four pods sit pending. On a cloud cluster the autoscaler would add a node and everyone lands; on this fleet, the night shift gets paged about stuck tenants while paid-for cores spin on nothing.

After: gang scheduling with minCount: 4. The scheduler evaluates all four gang pods against one snapshot, finds no valid placement, and binds none of them — the whole group waits in queue as one unit. Both tenant pods land in the untouched 7-CPU slots. Allocation still reads 84%, but now every allocated core does real work, zero pods sit pending, and the gang lands atomically the moment an earlier batch job finishes and frees a node. Nothing about the workload changed; the scheduler just stopped committing half a gang.

Now the second variant, because real fleets mix priorities: the same pool also runs latency-sensitive web tenants next to preemptible overnight batch. The old answer is hand-tuned PriorityClass values plus hope — and classic preemption evicts individual pods, which can tear one pod out of a running gang and wedge the other three.

Workload-aware preemption, merged into the GenericWorkload gate in v1.37, lets the scheduler preempt at group granularity. Set disruptionMode: {all: {}} on the batch PodGroup and a higher-priority web scale-up evicts the whole gang together — it requeues as a unit and relands atomically later — instead of orphaning survivors. Set disruptionMode: {single: {}} (the default) and individual pods stay independently preemptible, the old behavior.

v1.37 also fixed the v1.36 gap where default single-pod preemption ignored PodGroup disruption modes entirely, and added an authoritative preemptionPolicy field on the PodGroup itself behind the PodGroupPreemptionPolicy gate. For the overnight-batch-vs-web mix, that is the difference between "preemption wedges my gang" and "preemption is the bin-packing tool."

The opt-in checklist​

None of this turns on with a plain upgrade. Gang scheduling stays disabled by default in v1.37; you opt in per cluster. The checklist for a Cluster API-managed fleet:

1. Enable the GenericWorkload feature gate on both kube-apiserver and kube-scheduler. The apiserver must serve scheduling.k8s.io/v1beta1 (core Workload/PodGroup) and v1alpha3 (CompositePodGroup); the scheduler must run the PodGroup cycle and workload-aware preemption. If you manage scheduler configuration through a CAPI manifest, the flags live in your control-plane machine template or KubeSchedulerConfiguration drop-in:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
# Feature gates are passed as scheduler + apiserver flags in most
# CAPI bootstrap providers, e.g.:
#   --feature-gates=GenericWorkload=true[,PodGroupPreemptionPolicy=true]

2. Pin the API version to your Kubernetes minor version — they churned. v1.35 served v1alpha1 with workloadRef on pods; v1.36 replaced it wholesale with v1alpha2 and the schedulingGroup field; v1.37 promotes the core to v1beta1 while CompositePodGroup arrives at v1alpha3, and renames disruption modes (PodGroup→all, Pod→single). If you piloted any alpha, budget a manifest migration, not just a flag flip.

3. Wire up controllers instead of hand-writing PodGroups. Native management status by controller:

4. Label nodes for any topology constraints you declare. Topology-aware scheduling (single-level in v1.36, multi-level via CompositePodGroup in v1.37) resolves schedulingConstraints.topology keys against node labels. On a Hetzner fleet that means deciding your own topology vocabulary — rack, switch, datacenter-park — and applying it consistently, because there is no cloud provider to label zones for you.

5. Stay homogeneous for the first rollout. The PodGroup cycle guarantees finding a placement only for homogeneous groups without inter-pod affinity, anti-affinity, or topology-spread constraints. Heterogeneous gangs and groups with inter-pod dependencies may fail to place even when capacity exists. Keep your first gang workloads identical-pod jobs.

Honest limits before you flip the gate​

Beta is not GA, and the alpha edges are exactly where a small team gets hurt. The heterogeneous-group placement caveat above is the big one: if your gang mixes pod shapes or leans on affinity rules, test placement behavior under fragmentation before trusting it. CompositePodGroup hierarchies, multi-level topology, and Job integration depth all remain alpha, so a driver-plus-workers shape like a Spark or vLLM deployment still lives partly on alpha APIs with breaking-change history across all three releases.

And the incumbents have not stood still. Volcano brings years of queueing, quota, and fair-share machinery the in-tree work deliberately does not replicate yet, plus a June 2026 Headlamp plugin that makes gang and queue state visible in the same console as the Cluster API plugin. Kueue's maintainers have an open RFC to adopt the in-tree PodGroup rather than fight it — a sign the ecosystem expects convergence, but convergence is not arrival. If you need fair-share queues across tenants today, you still want Kueue or Volcano in front; if you need atomic placement of a single tenant's gang without a second system, v1.37 is the first release where the tree alone answers.

One more operational note: native PodGroup queueing in v1.37 changes scheduling-queue observability — the group object queues instead of each member pod. Dashboards counting pending pods will undercount queued gang members until they learn to read PodGroup status. Update the monitor before the rollout, not after the first confusing page.

What a small fleet should actually do​

If your Cluster API fleet is still on pre-1.35 Kubernetes, workload-aware scheduling is a reason to keep your upgrade runbook current, not to rush one: track the v1.37 Beta as the release where the core API stabilizes, pilot gang scheduling on a staging cluster with a homogeneous batch job, and migrate any alpha-era manifests forward.

If you already run Volcano or Kueue for batch tenants, stay — and watch the Kueue PodGroup-adoption RFC, because the migration story will likely come from that direction. If you run neither and your fixed node count keeps wedging on half-scheduled gangs, v1.37 is the first release where the fix is a feature gate instead of a second scheduler.

Fixed capacity punishes partial scheduling and rewards atomic placement. The scheduler finally sees the workload. Turn it on deliberately, measure the pending-pod count before and after, and keep the nodes you already paid for.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.