Kubernetes 1.36 Lets You Resize a Job Before It Starts: The Queue-Then-Resize Pattern a Build Queue Needed

Your build-queue controller just admitted a Job asking for 4 vCPU and 8Gi to compile a large monorepo image. The shared builder node pool has 2 vCPU and 4Gi free — two other builds are already running on it. Before Kubernetes v1.36, you had exactly two options: leave the Job Pending and hope capacity frees up, or delete it and recreate a smaller one, severing the Job UID that your webhook, your commit-status check, and your build-log tail were all keyed off of. As of v1.36, there's a third option: patch the suspended Job's pod template down to what's actually free, then unsuspend it — same Job, same UID, same history, just resized before a single pod exists.

That's MutablePodResourcesForSuspendedJobs, and it graduated from alpha (v1.35) to beta, enabled by default, in v1.36. It's a narrow change — a few fields in the Job API that used to be frozen at creation time can now be edited while the Job is suspended — but it closes a real gap for anything that queues Kubernetes Jobs against capacity it doesn't fully control, which describes a self-hosted PaaS's build pipeline as directly as it describes an ML training queue.

What Actually Changed​

Before v1.36, a Job's pod template was immutable the moment the API server admitted it — with spec.suspend as basically the only field a controller could still touch. If Kueue or a custom queue controller decided a suspended Job should run with different resources than it was created with, the only move was to delete the Job and recreate it, which meant losing whatever status, annotations, or owner-reference history was tracked against the original object.

KEP #5440, now beta in v1.36, relaxes that immutability for exactly four fields, and exactly one Job state:

• spec.template.spec.containers[*].resources.requests and .limits
• spec.template.spec.initContainers[*].resources.requests and .limits

That covers CPU, memory, GPUs, and vendor extended resources (anything shaped like example-hardware-vendor.com/gpu) — both requests and limits, on both regular and init containers. Standard validation still applies (limits still have to be ≥ requests), and no new API type was introduced; this is relaxed validation on the existing batch/v1 Job, not a schema change.

The mutation is only legal when:

1. spec.suspend: true, and
2. for a Job that was previously running and got suspended, status.active == 0 — every Pod it owns has actually terminated.

That second gate is the whole safety argument, and it's worth stating plainly rather than leaving it implied: the feature never lets you rewrite the resource footprint of a Pod that's currently running. A queue controller can only mutate a Job in the window where zero Pods exist for it — nothing to disrupt, nothing to OOM-kill mid-flight, nothing for the kubelet to reconcile against live cgroups. That's a narrower, safer operation than in-place pod resize (GA since v1.35), which patches a container's cgroup limits while it's actively serving traffic and has to handle partial failure, resizePolicy restart semantics, and a PodResizePending deferred state. Mutable suspended-Job resources sidesteps all of that by construction — there's no running container to fail to resize, because resizing happens before the container exists. That's also why this shipped as relaxed validation rather than a new resize subresource: nothing runtime is being touched, only a spec the scheduler hasn't acted on yet.

It's also worth being specific about what delete-and-recreate actually cost, because "loses metadata" undersells it. A Job's UID is what a JobSet, a CronJob's active-Jobs list, and any ownerReference pointing at it are keyed on — delete and recreate, and every one of those relationships has to be rebuilt against a new UID rather than simply continuing. status.startTime, status.conditions, and the Job's accumulated Pod failure/backoff history reset to zero. If a webhook or a commit-status check was watching that specific object, it has to detect the deletion, find the replacement, and re-subscribe — a race a queue controller has to write and test defensively, not something Kubernetes hands you for free. Beta-in-1.36 mutation removes the need to write that reconciliation logic at all: the object a controller is watching before the resize is the same object it's watching after.

The Patch, Concretely​

Here's the shape of it against the build-Job example above — a queue controller (or, until you've wired one up, kubectl by hand) patches the suspended Job's resources, then flips suspend to false:

# Job is suspended, requesting more than the pool currently has free
kubectl get job build-a1b2c3d -o jsonpath='{.spec.template.spec.containers[0].resources}'
# {"requests":{"cpu":"4","memory":"8Gi"},"limits":{"cpu":"4","memory":"8Gi"}}
 
# Queue controller checks free capacity, patches the suspended Job down to fit
kubectl patch job build-a1b2c3d --type=strategic -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "buildkit",
          "resources": {
            "requests": {"cpu": "2", "memory": "4Gi"},
            "limits":   {"cpu": "2", "memory": "4Gi"}
          }
        }]
      }
    }
  }
}'
 
# Resume it — same Job object, same UID, same annotations/labels
kubectl patch job build-a1b2c3d -p '{"spec":{"suspend":false}}'

Nothing about build-a1b2c3d's identity changed. The commit SHA in its annotations, the label your webhook receiver matches on, the ownerReference back to whatever CRD represents "this deploy," the accumulated status history — all of it survives, because the object was never deleted. The only thing that moved is the number the scheduler checks against node capacity. The build takes longer at 2 vCPU than it would have at 4 — that's the real tradeoff, not a free lunch — but it starts now instead of sitting Pending indefinitely or requiring a human to notice and manually re-run it smaller.

What This Doesn't Solve​

This is capacity-aware admission, not capacity creation. Three limits worth being explicit about before wiring this into anything:

• It's still bounded by the node pool's real ceiling. If nothing in the fleet has 2 vCPU/4Gi free either, patching a suspended Job down further doesn't materialize capacity — a Cluster Autoscaler, or on a Cluster-API-provider-Hetzner fleet, CAPH provisioning another Hetzner machine and joining it to the pool, is still the only way to grow the ceiling itself.
• There's a real floor on how far you can shrink a job and expect it to succeed. A build that genuinely needs 6Gi to link a large binary will OOM at 4Gi regardless of how gracefully you resized it down — this feature makes the patch safe, not the number you pick. A queue controller that blindly halves requests to fit whatever's free will trade a Pending Job for a OOMKilled one.
• It only applies to Jobs, and only while suspended. A long-running Deployment's Pods still go through in-place resize (or restart) if they need to change size after they're already up; this KEP doesn't touch that path at all.

Wiring It Into a Build-Queue Controller​

The mechanical loop a queue controller needs is small: watch incoming suspended Jobs, compare each one's requested resources against currently-free node-pool capacity, patch down (never below whatever minimum you've established the workload actually needs), then unsuspend. Kueue is the reference implementation of this pattern today — its ClusterQueue/LocalQueue admission logic already tracks per-flavor free capacity, and as of v1.36 it can use this KEP to right-size a Job in place instead of falling back to delete-and-recreate when an exact-fit slot isn't available. Kueue layers its own ProvisioningRequest integration on top for the case where nothing fits at all — that's the trigger for a Cluster Autoscaler to add a node, which is a separate mechanism from the resize this KEP ships; the two compose (resize first if a smaller footprint fits now, provision new capacity if nothing does) rather than replace each other.

A bespoke build-queue controller on a Cluster-API-managed fleet doesn't need to adopt Kueue wholesale to get the same behavior — the primitive is just a client-go Patch call against the Job's spec.template.spec.containers[*].resources while suspend: true, gated on the same free-capacity check the controller already has to do to decide whether to admit a Job at all. The beta feature gate (MutablePodResourcesForSuspendedJobs) is on by default in 1.36; the only real prerequisite is a control plane already running 1.36, since 1.35 shipped it alpha-only, off by default, and an older client attempting the same patch against a pre-1.35 API server will simply get the old validation error back — the mutation only becomes legal once the API server itself has upgraded, independent of what version any given controller binary is running.

The pattern generalizes past builds: any batch workload a self-hosted PaaS queues against capacity it doesn't fully control up front — scheduled jobs, one-off migrations, AI-agent sandbox provisioning runs — gets the same option. Right-size to what's free and start now, instead of waiting for an exact-fit slot that may never open.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, with a build pipeline that's a real Kubernetes Job on a fleet you control end to end. Star the repo on GitHub or deploy your first app today.