Gang Scheduling Is Beta in Kubernetes 1.37 — and Off: The Opt-In Checklist for GPU Node Pools

One headline about Kubernetes 1.37 says gang scheduling "cuts GPU idle costs by default." The feature documentation says, in its first line: Beta since Kubernetes v1.37; disabled by default. Both are about the same release, and only one of them describes your cluster after you upgrade. (The article's own body walks the headline back — "disabled by default… does not turn on with a standard upgrade" — which is exactly the gap this post is about.)

This gap between headline and default matters more than usual because the waste it promises to fix is enormous. Cast AI's 2026 State of Kubernetes Optimization Report, published October 6 and drawn from roughly 23,000 clusters, measured average GPU utilization at 5% — 95% of provisioned accelerator capacity sitting idle — with CPU utilization sliding backwards to 8% from 10% a year earlier. A separate ArcCompute dataset covering January 2025 to April 2026 landed on the same number: about 95% idle.

Against that backdrop, "all-or-nothing placement for distributed training jobs" reads like a rescue. It can be, but a bare upgrade to v1.37 does not change a single scheduling decision on your cluster. The rescue is opt-in, per component, and one stale flag from the alpha era will break your control plane if you leave it in.

This post is the fine print, executable: what actually graduated in Garhwal, the exact checklist that turns the beta into working gang scheduling, what a bare upgrade gives you (nothing, by design), and the Cluster API version of the rollout for a fleet planning GPU-backed pools on owned nodes.

What actually graduated in Garhwal​

Kubernetes v1.37 "Garhwal," released August 26, 2026, ships 67 enhancements: 16 stable, 23 beta, 27 alpha, and one removal. The scheduling headline is KEP-4671: native gang scheduling graduates to beta through the Workload and PodGroup APIs, so a defined group of pods is placed all-or-nothing — if the cluster cannot fit the group, or at least its minCount, none of its pods bind, instead of half a training job landing and deadlocking while its workers wait for siblings that never arrive.

The nouns you need for the rest of this post:

Two things that did not graduate: the CompositePodGroup API for hierarchical, multi-level gangs (alpha, scheduling.k8s.io/v1alpha3), and topology-aware scheduling (alpha). The v1.38 roadmap targets GA for the core Workload/PodGroup APIs and beta for those two, with Kueue expected to lean on this machinery as its underlying gang-scheduling engine over time.

The opt-in checklist​

Here is the core deliverable: everything between "we run v1.37" and "a gang-scheduled pod lands." The upstream getting-started section states the premise plainly — beta and alpha features here are disabled by default and require manual enablement — and then lists six gates, every one of them default-false.

1. Enable GenericWorkload on three control-plane components. This single beta gate now covers gang scheduling and workload-aware preemption (the separate WorkloadAwarePreemption gate was merged into it). Set --feature-gates=GenericWorkload=true on the kube-apiserver, kube-scheduler, and kube-controller-manager, and roll the control plane. Once the gate is on, the built-in GangScheduling scheduler plugin handles PodGroups — this is the point of the graduation: no Volcano, no Kueue, no scheduler-plugins build just to get all-or-nothing placement.

2. Confirm the scheduling.k8s.io/v1beta1 API is served. Update manifests to the beta API version, then verify before creating anything:

kubectl api-resources | grep -i podgroup
kubectl get --raw /apis/scheduling.k8s.io/v1beta1 | head -c 300

If your distribution does not serve the group, the apiserver needs --runtime-config=scheduling.k8s.io/v1beta1=true as well. Unlike the old coscheduling plugin from scheduler-plugins, there is no CRD to install — PodGroup is a built-in type — but "built-in" and "enabled" are different words, and only one of them is true after an upgrade.

3. Delete the standalone GangScheduling gate from old configs. The alpha-era GangScheduling feature gate was removed upstream and consolidated into GenericWorkload. Any kind config, kubeadm manifest, or scheduler args file that still sets GangScheduling=true is now setting a gate that does not exist, and components fail on unrecognized gates — the Kueue project hit exactly this when its test configs started failing with unrecognized feature gate: GangScheduling. Grep your fleet's configs for it before you roll; this is the one step that punishes the careful early adopter who enabled the alpha and then forgot about it.

4. Create a PodGroup and join pods to it. Pods opt in individually through the immutable spec.schedulingGroup.podGroupName field:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-worker-0
  namespace: default
spec:
  schedulingPolicy:
    gang:
      minCount: 4

Then join each member pod to the group:

apiVersion: v1
kind: Pod
metadata:
  name: worker-0
  namespace: default
spec:
  schedulingGroup:
    podGroupName: training-worker-0
  containers:
    - name: ml-worker
      image: training:v1

5. Or declare it on the Job — with one more gate. v1.37 adds .spec.scheduling to batch Jobs, so the Job controller compiles your policy into the Workload and PodGroup objects itself via the shared workloadbuilder library:

spec:
  parallelism: 8
  completions: 8
  scheduling:
    schedulingPolicy:
      gang: {}   # minCount omitted: defaults to parallelism (8)
    disruptionMode:
      all: {}

The fine print inside the fine print: Job-controller integration sits behind the alpha WorkloadWithJob gate (apiserver and controller-manager), and .spec.scheduling is immutable once set — except schedulingPolicy.gang.minCount, which you can update to resize a running gang.

6. Verify the gang, not just the version. kubectl version telling you v1.37 proves nothing about scheduling behavior. Check the PodGroup's conditions and watch placement actually gate on quorum:

kubectl get podgroup training-worker-0 -o jsonpath='{.status.conditions}'

Create the PodGroup with minCount: 4 but only two member pods, and confirm both sit unscheduled until the remaining two exist — that held-in-PreEnqueue behavior is the feature working. If two pods bind immediately, you are reading the headline, not running the feature.

What a bare upgrade actually gives you​

This section exists so you can calibrate the "nothing changed" support ticket you will otherwise file against yourself. If .spec.scheduling is omitted from a Job, or schedulingPolicy is omitted within it, the Job takes the basic policy, which behaves exactly like standard Job scheduling today. The controller still creates a Basic Workload and PodGroup behind the scenes, but with no minCount gate enforced — placement is pod-by-pod, partial gangs land, and your GPU bins pack exactly as well as they did on v1.36.

That is the designed default, not a bug: upgrading Kubernetes alone does not automatically change cluster scheduling behavior. Treating a release headline as an enabled feature is precisely how a fleet upgrades to v1.37 and then discovers, weeks later during the first contested GPU window, that nothing about its packing changed. Detect the state early — a PodGroup stuck without conditions, pods binding below quorum — rather than during an incident.

The Cluster API fleet version​

For a Cluster API fleet on owned nodes, the checklist above compresses into one template change and one rollout discipline. The gates are control-plane flags, so on a kubeadm-based fleet (CAPH included) they live in the KubeadmControlPlane template's clusterConfiguration — apiServer, scheduler, and controllerManager extraArgs — which means the v1.37 upgrade PR and the gang-scheduling enablement are separable diffs, and should be reviewed as such.

Roll them in this order: first the Kubernetes version bump to v1.37 with behavior unchanged, confirming the fleet is healthy on the new binaries; then the gate diff, which rolls the control-plane machines and changes scheduling semantics only for workloads that opt in via PodGroups. Keep the two diffs separate because their blast radii are different — one touches every pod's kubelet and runtime path, the other touches only gangs — and because the rollback stories differ: reverting a version bump re-images machines, while reverting a gate is a flag flip. Add the stale-GangScheduling-gate grep from step 3 to the upgrade runbook permanently; every future release that consolidates a gate will try to bite the same way.

One scoping note for fleets without GPUs today: none of this requires GPU hardware to validate. minCount quorum, PreEnqueue hold behavior, and workload-aware preemption are all exercisable with CPU-only Jobs on a staging management cluster, which is exactly where the checklist should be rehearsed before the first GPU node pool arrives.

The second gate GPU pools need​

If your GPU story runs through Dynamic Resource Allocation rather than the legacy nvidia.com/gpu integer hack, gang scheduling alone is half the picture. The DRAWorkloadResourceClaims gate — beta in v1.37, also off by default — lets a ResourceClaim be replicated and reserved for an entire PodGroup and shared by all member pods, which is the shape GPU-backed gangs actually want: one claim for the gang's accelerators, not N per-pod claims that can fragment.

It needs enabling on four components, not three: apiserver, controller-manager, scheduler, and kubelet. And v1.37 fixed a sharp edge worth knowing before you toggle it: previously, if a pod's claim referenced a template matching its PodGroup's claim while the gate was disabled, Kubernetes created a per-pod ResourceClaim — a flood of claims from one template, potentially exhausting DRA resources. In v1.37, no claim is created at all in that state. The failure mode went from "silently expensive" to "visibly nothing," which is the right direction, but it is another behavior that reads as "broken" until you know which gate you forgot.

What to wait for​

Not everything in the announcement is ready for a fleet runbook. CompositePodGroup (hierarchical gangs for multi-component workloads like replicated training, with minGroupCount across child groups) is alpha and needs both the CompositePodGroup and TopologyAwareWorkloadScheduling gates plus the v1alpha3 API. PodGroup-level preemptionPolicy is alpha behind PodGroupPreemptionPolicy. The v1alpha2-to-v1alpha3 transition also carried breaking changes around disruptionMode, so anything carried forward from 1.36-era testing needs its manifests re-checked, not just re-applied.

The adoption judgment for a self-hosted fleet: enable GenericWorkload on staging now and exercise the Job-integrated path (with WorkloadWithJob) against CPU workloads, so the machinery is proven before GPU pools make it load-bearing. Leave CompositePodGroup and topology-aware scheduling on the roadmap watchlist for the v1.38 beta graduations. And keep one eye on Kueue: upstream's stated long-term direction is Kueue leveraging this native machinery as its gang-scheduling engine, which means work you do today to express gangs as PodGroups compounds rather than forks.

The pattern here generalizes beyond one release. Kubernetes increasingly ships its most valuable scheduling behavior as disabled-by-default betas — the code rides the upgrade train, but the behavior waits for a flag. The operators who benefit are the ones who read the getting-started section, not just the headline. v1.37's gang scheduling really can cut GPU idle costs. It just can't do it by default.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.