Skip to main content

Kubernetes 1.36 Ships Gang Scheduling In-Tree: What It Actually Buys an AI-Agent Batch Fan-Out

8 min readDora NodaDora Noda
Share
On this page

An AI agent kicks off a five-worker fan-out through your platform's MCP surface — a batch job that only makes progress if all five pods land and start together. Three land, two don't fit, and the job burns compute for twenty minutes going nowhere: three pods block on IPC calls to workers that were never scheduled, and nothing tells the scheduler to give up and retry as a group instead of pod-by-pod. That failure mode has a name — gang scheduling — and until now, fixing it in-tree meant either accepting it or bolting on Volcano or Kueue as a second scheduler.

Kubernetes v1.36, released April 22, 2026, changes that math. It ships a PodGroup and Workload API directly in the core scheduler, behind five alpha feature gates. This post is the concrete rundown: what each gate does, what a Job actually looks like once it's gang-scheduled, what this replaces a third-party scheduler for on a self-hosted PaaS running its own Hetzner nodes — and, just as importantly, how far "alpha" still is from something you'd trust with tenant workloads today.

What Ships in 1.36, Concretely

Five feature gates, all alpha, all off by default. This is the whole surface area — everything else in this post is what turning them on gets you.

Feature gateKEPWhat it does
GenericWorkloadKEP-4671Base gate — enables the Workload and PodGroup API objects on the apiserver and scheduler
GangSchedulingKEP-4671All-or-nothing placement: a PodGroup with minCount: N binds all N pods together or none
WorkloadWithJobKEP-5547The stock Job controller auto-creates the Workload/PodGroup and stamps .spec.schedulingGroup on every pod it spawns
TopologyAwareWorkloadSchedulingKEP-5732Co-locates a PodGroup's pods within one topology domain (a rack, a block) instead of scattering them
WorkloadAwarePreemptionKEP-5710Preemption evicts a whole low-priority PodGroup to make room for a high-priority one, instead of picking off individual pods

The API itself splits into two objects — a change from 1.35's design, via KEP-5832. Workload is a static template: it defines a podGroupTemplates list with a scheduling policy (gang.minCount, or basic for ordinary behavior) but the scheduler never has to parse it directly. PodGroup is the runtime object a controller instantiates from that template — it's the thing the scheduler actually watches, and it carries a status.conditions field the scheduler flips once the group is bound. Separating the two means the scheduler's hot path only ever touches the lightweight runtime object, not a full template definition, which is why the split exists at all rather than being an API nicety.

The PodGroup/Workload Split, and What WorkloadWithJob Automates

Here's the mechanism, worked through for the exact scenario in the hook — a five-pod agent-triggered worker fan-out.

Without WorkloadWithJob, gang-scheduling a Job means hand-authoring both objects yourself: a Workload with a gang.minCount: 5 policy, a PodGroup instantiated from it, and a pod template that sets spec.schedulingGroup on every pod to point at that PodGroup. That's real plumbing a platform controller would have to own.

With WorkloadWithJob on, the Job controller does all of it for you:

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: agent-worker-fanout
spec:
  parallelism: 5
  completions: 5
  completionMode: Indexed
  template:
    spec:
      containers:
        - name: worker
          image: registry.bex.co/acme/worker:pr-902
      restartPolicy: Never

That's an ordinary Job — nothing gang-scheduling-specific in the spec. When the gate is on, the controller creates a Workload and a PodGroup owned by the Job, sets minCount to match parallelism, and stamps spec.schedulingGroup referencing that PodGroup onto all five pods it spawns. The scheduler's GangScheduling plugin then holds every pod at the Permit barrier until all five have cleared admission, and binds them together — or, if only three fit on current capacity, holds all five back rather than starting three and stranding the other two mid-fan-out. Delete the Job and the generated Workload/PodGroup garbage-collect with it, same as any owned object.

The scope is narrower than "any Job," though, and it's worth stating plainly: KEP-5547 currently covers only static, indexed, fully-parallel Jobs — the shape a worker fan-out already is. An elastic Job that changes its parallelism mid-run, or a batch shape driven by a controller other than the stock Job controller (a CronJob-spawned Job, JobSet, KubeRay), isn't covered by this integration path yet. For those, a platform still has to instantiate Workload/PodGroup by hand, the way you'd have had to for every Job before WorkloadWithJob existed.

Turning any of this on takes two flags, set the same way on both components: --feature-gates=GenericWorkload=true on kube-apiserver (so the scheduling.k8s.io/v1alpha2 API group is actually served) and the same gate plus GangScheduling=true on kube-scheduler. WorkloadWithJob is a separate gate on kube-controller-manager, since that's the component that runs the Job controller. Each gate is independent and additive — GangScheduling alone gets you the all-or-nothing binding behavior; WorkloadWithJob on top removes the need to hand-author the Workload/PodGroup objects; TopologyAwareWorkloadScheduling and WorkloadAwarePreemption are opt-in refinements on top of that base, not prerequisites for it.

TopologyAwareWorkloadScheduling is worth a concrete note, because "topology-aware" undersells what it actually targets: KEP-5732 adds TopologyConstraints and DRAConstraints fields to the Workload API so a PodGroup's pods land within one labeled domain — topology.kubernetes.io/rack, for instance — instead of scattering across the cluster and paying cross-rack network latency on every worker-to-worker call. For a self-hosted PaaS running its own Hetzner fleet, that's the primitive that would eventually let a platform express "keep this fan-out on nodes that share a rack" without hand-rolling pod affinity rules per Job. It's alpha in 1.36 with Beta targeted for 1.37, same cadence as the rest of this feature set.

What This Buys — and Doesn't — Versus Volcano or Kueue

The honest framing for a self-hosted PaaS: this doesn't replace Volcano or Kueue, but it does shrink the reason you'd reach for either one for basic all-or-nothing placement.

CapabilityIn-tree (1.36, alpha)KueueVolcano
All-or-nothing pod-group placementYes — GangSchedulingYesYes
Multi-tenant fair-share / hierarchical quotaNo — explicit non-goal, delegatedYes — the core featurePartial (queue-based)
PyTorchJob / MPIJob / RayJob-native integrationNoYesPartial
HPC topology awareness (NVLink, NUMA)Partial — TopologyAwareWorkloadScheduling covers rack/block-level, not NUMA/NVLinkNoYes
Runs as a second scheduler / extra componentNo — lives in kube-schedulerYesYes
MaturityAlpha, all gates off by defaultStable, widely deployedStable, widely deployed

For a fan-out that just needs "all N pods land together or none do" — the case that matters most for an AI-agent worker fan-out on owned Hetzner capacity — 1.36 makes that a scheduler-native property instead of a reason to stand up and operate a second scheduling system. That's real: fewer moving parts, one less component with its own upgrade cadence and failure modes on a Cluster API fleet you already run. But KEP-4671 states outright that fairness and multi-queue scheduling are non-goals, left to Kueue and Volcano on purpose. The moment a platform needs to arbitrate GPU capacity fairly across multiple tenants' batch jobs, or wants native PyTorchJob/RayJob integration, in-tree gang scheduling alone doesn't cover it — Kueue still owns that layer, and would sit on top of, not instead of, what 1.36 ships.

How Far Alpha Actually Is From Safe for Tenant Workloads

This is the part a feature announcement won't spell out as bluntly, and it's the part that actually determines whether to touch this today.

All five gates are alpha, and alpha in Kubernetes means no compatibility guarantee across releases — the API shape, field names, and defaults can change or disappear between 1.36 and 1.37 without a deprecation cycle. Enabling GenericWorkload on a shared apiserver commits you to re-testing against every subsequent minor release until this graduates.

The batch scheduling improvement is still incomplete. 1.35's alpha implementation gang-schedules through two barrier extension points (PreEnqueue, Permit) checked per-pod rather than a true batch cycle that evaluates a whole group's feasibility in one pass. A dedicated Workload Scheduling Cycle that processes an entire PodGroup atomically was targeted for Beta in 1.36 but was deferred to 1.37 — so the scheduling logic underneath these gates today is a real improvement over nothing, but not yet the fully batched implementation the design settles on.

WorkloadWithJob's scope is narrow, as covered above — static, indexed, fully-parallel Jobs only. Anything else still needs hand-rolled Workload/PodGroup objects.

Preemption is separately alpha and layered on top. WorkloadAwarePreemption depends on GangScheduling already being enabled, and getting group-preemption wrong — evicting a whole PodGroup incorrectly — is a worse failure mode than mis-preempting a single pod, since it can cascade into re-preempting other groups to make room. This is not the gate to flip on first.

No fairness or quota, by design — covered above, but worth repeating here because it's the gap most likely to bite a multi-tenant platform specifically: gang scheduling alone doesn't stop one tenant's fan-out from starving another's, it only guarantees a single fan-out lands atomically or not at all.

The practical read for a self-hosted PaaS: this is worth running in a dev or staging Cluster API cluster today, against non-production batch work, specifically to build familiarity before it graduates — not something to flip on for tenant-facing agent batch jobs yet. The 1.37 beta cycle, expected to land the completed Workload Scheduling Cycle, is the milestone that actually changes that recommendation.


Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with agent-triggered batch fan-outs running on the same Cluster API fleet you already operate rather than a third-party scheduler bolted on top. Star the repo on GitHub or deploy your first app today.

Related articles

Give your agents a chain backend

Autonomous agents hit RPC endpoints very differently than people do. See what bex router handles on their behalf.

Read the agents guide