Skip to main content
bex.co
419
Blog

Kubernetes Has Scheduled GPUs as Integers Since 2017 — What KAI Scheduler and Grove Actually Change for a Fleet Without Multi-GPU Nodes

10 min readDora NodaDora Noda
Self-Hosting
PaaS
Infrastructure
AI
Engineering
Share
On this page

A pod asks for 2 GPUs. The cluster has 2 GPUs free — added up across every node. The pod sits Pending anyway, because those 2 free GPUs are on two different nodes with one slot each, and nothing between the scheduler and the kubelet has ever tracked which node a GPU actually lives on beyond a bare integer. This has been true since Kubernetes 1.8 shipped the device plugin model in 2017, and it's a large part of why Cast AI's 2026 State of Kubernetes Optimization report — drawn from tens of thousands of production clusters — found average GPU utilization sitting at just 5%, against 8% for CPU and 20% for memory.

The short answer for a fleet like bex's: Hetzner's dedicated GPU servers are all single-GPU-per-node — there's no multi-GPU box to buy — so the textbook fragmentation story above literally can't happen on this hardware. The real exposure shows up one layer down: gang-scheduling deadlock when a tensor-parallel workload needs several GPUs at once across several separate nodes, and MIG-slice fragmentation inside the one card that supports hardware partitioning. CAPH's default scheduler config has zero handling for either case — not incomplete, genuinely absent. KAI Scheduler, NVIDIA's open-source Kubernetes scheduler, is worth adding now to close that gap. Grove, NVIDIA's newer topology-aware placement API, isn't — not yet, not at this fleet size. Here's the reasoning behind both calls.

The 2017 contract, and where it breaks​

The device plugin framework landed in Kubernetes 1.8 to solve a real problem: the scheduler had no concept of specialized hardware at all. The fix was a clean, minimal contract. A vendor daemon — NVIDIA's device plugin, running as a DaemonSet on every GPU node — reports a count to the kubelet, which advertises it to the API server as an "extended resource": nvidia.com/gpu: 4. The scheduler subtracts requested integers from available integers, the same arithmetic it already does for CPU cores. Kubernetes' own docs are explicit that extended resources "are only supported as integer resources and cannot be overcommitted" — no fractions, no partial allocation, full stop.

That assumption — any two GPUs are interchangeable, and a node's free capacity is fully described by one integer — held fine in 2017, when clusters were homogeneous and a "GPU workload" meant one training job claiming a whole 8-GPU node for days. It stopped holding once inference workloads showed up wanting one GPU each, sharing nodes with training jobs wanting all eight, on hardware mixing three or four different GPU models. The scheduler still only sees the count.

Here's the canonical failure mode on that kind of cluster. Take an 8-GPU-per-node fleet, common on hyperscaler and on-prem multi-GPU boxes (a DGX-class node, or a cloud instance type with 8 attached accelerators). Seven nodes are running training jobs at 7-of-8 GPUs each — one card free per node, seven free GPUs total, spread one-per-node. A new job needs 2 GPUs on the same node for NVLink-speed collective communication. None of the seven nodes has 2 free. The pod stays Pending even though the cluster-wide free count is 7. Worse, the cluster autoscaler doesn't step in, because autoscaling triggers on a node being short, and no single node is short — each one reports exactly the free capacity it has. The fragmentation is a property of the distribution across nodes, and integer-count scheduling has no way to represent that, let alone optimize for it. Industry write-ups on device-plugin clusters converge on 20-30% GPU idle time attributable to exactly this pattern.

What KAI Scheduler and Grove actually do​

Two 2026-relevant tools attack this gap directly, alongside Dynamic Resource Allocation, the structured-device-claim framework that graduated to GA in Kubernetes 1.34 (March 2026) and gives drivers a way to publish per-GPU attributes — memory, MIG profile, NVLink topology — instead of a bare count.

KAI Scheduler started life inside Run:ai's commercial platform; NVIDIA acquired Run:ai in late 2024 and open-sourced the scheduler in April 2025 under Apache 2.0, and it now sits in the CNCF Sandbox. Its core mechanism is bin-packing rather than spread-scheduling: it sorts the cluster's topology tree by allocatable slots ascending, so a job is placed into the tightest domain that fits it, leaving contiguous blocks free instead of thin slices on every node. On top of that it adds workload consolidation (reallocating running jobs to further reduce fragmentation), gang scheduling (all-or-nothing placement for a job's pods, so a multi-pod workload doesn't partially land and hold resources hostage), and hierarchical queues with quotas — the multi-tenant primitive a device-plugin-only cluster has no equivalent for.

Grove is newer and narrower: an open-source Kubernetes API — PodCliqueSet, PodClique, PodCliqueScalingGroup, PodGang — purpose-built for multi-node inference topologies, open-sourced at KubeCon Europe 2026. It expresses gang scheduling with startup-order dependencies (a prefill pod group before a decode group, say) plus network-topology-aware placement — packing gang members onto GPUs that share a fast interconnect, and spreading them across failure domains when availability matters more than locality. KAI Scheduler and Grove are designed to compose: KAI does the bin-packing and queueing, Grove expresses the gang shape and topology constraints on top.

What this means for bex's actual fleet​

Here's the piece the canonical fragmentation story skips past: it assumes multi-GPU nodes. bex runs on Hetzner's dedicated GPU line — the GEX44 (one RTX 4000 SFF Ada, 20GB) and the GEX131 (one RTX PRO 6000 Blackwell Max-Q, 96GB) — and neither SKU, nor any other GEX-series box Hetzner sells, has more than one GPU per node. There is no 8-GPU node to fragment. The "seven nodes at 7-of-8, nobody has 2 free" scenario has no host to occur on.

That doesn't mean the underlying problem disappears — it moves down a layer, to two places a single-GPU-per-node fleet actually has exposure:

MIG-slice fragmentation, inside one card. The GEX131's Blackwell card supports NVIDIA Multi-Instance GPU, up to four hardware-isolated slices per card; the GEX44's Ada-generation card doesn't support MIG at all. A GEX131 partitioned into four MIG instances is functionally a 4-slot node — and a scheduler with no bin-packing awareness can strand it exactly the way an 8-GPU node gets stranded: three small slices claimed by three tenants in a way that leaves capacity free but not shaped for the next pending request.

Cross-node gang-scheduling deadlock, across many. This is the sharper exposure, because it doesn't require MIG at all. Any tenant workload that needs tensor- or pipeline-parallel serving — a model too large for one GEX131's 96GB, split across two or more GPU nodes — needs several pods scheduled simultaneously on separate single-GPU boxes, or the job can't run correctly. Vanilla kube-scheduler has no gang-scheduling primitive: it schedules pods independently, so it's entirely possible for 2 of 4 required pods to land, claim their GPUs, and then sit there holding hardware while the remaining 2 stay Pending behind other queued work — a resource-holding deadlock, not a fragmentation percentage. Tenant AI-agent sandboxes, by contrast, are mostly single-GPU-per-pod and bursty — they consume the same scarce single-GPU inventory but don't hit this failure mode, since there's no multi-node coordination to deadlock on.

CAPH's default vs. hand-rolled vs. KAI Scheduler — the actual install delta​

The TODO framing this piece answers asks how far off-the-shelf tooling is from a CAPH-provisioned fleet's default scheduler config, versus a platform hand-rolling its own affinity rules. Three concrete points on that spectrum:

CAPH's default. Cluster API Provider Hetzner provisions the bare-metal host and runs kubeadm join — it has no opinion on the scheduler at all. Whatever lands on the cluster is vanilla kube-scheduler, and GPU-awareness arrives separately, after the node joins, when the NVIDIA GPU Operator installs the device plugin and writes node labels. Neither piece does bin-packing, gang scheduling, or MIG-shape awareness. This isn't "CAPH ships an incomplete GPU scheduler" — CAPH ships no GPU scheduler; the workload-placement layer discussed in an earlier post on deploying vLLM as a git-push app — a MachineDeployment per GPU SKU, a nvidia.com/gpu.product node-affinity rule, a Kueue ResourceFlavor per SKU — is entirely something the platform has to add itself.

Hand-rolled affinity (the current approach). That existing setup handles single-GPU, single-tenant-per-pod placement correctly and cheaply: a pod asking for one GPU on a given SKU lands on a node with that SKU, full stop. It has no answer for gang scheduling, because a MachineDeployment-and-label scheme has no concept of "these N pods must land together or not at all" — that's a scheduler-level primitive, not a provisioning-level one.

KAI Scheduler as a secondary scheduler. Installing KAI doesn't replace any of the above — it's a Helm chart deployed alongside the default scheduler, and GPU workload pods opt in with schedulerName: kai-scheduler in their spec while everything else keeps using the default. There's no CAPH-specific integration to write or wait for; it's a day-2 cluster addition, the same shape as installing the GPU Operator itself. That's the honest measure of "how far this tooling is from default": not close, but not far either — one Helm install and one field in the pod spec away, because the gap CAPH leaves is total rather than partial.

What's worth adopting now — and what to skip​

For a fleet at bex's current scale — a handful of GPU-bearing Hetzner nodes across one or two locations — the recommendation splits three ways:

  • KAI Scheduler: adopt now. It directly closes the two real exposures above — gang scheduling for tensor-parallel tenant workloads, and MIG-aware bin-packing on the GEX131. It's Apache 2.0, drops in as a secondary scheduler with no cluster-autoscaler dependency (which matters on a fixed-size owned fleet with no autoscaler to begin with), and the install cost is one Helm release.
  • Grove: skip for now. Its differentiator is network-topology-aware placement — packing gang members onto GPUs sharing a fast interconnect, spreading across racks for availability. That's a real problem at hundreds of GPUs spread across many racks and switches. It isn't a problem across a handful of Hetzner boxes in one or two datacenters, where every node is already roughly equidistant from every other. Revisit once the fleet's GPU node count and topology actually justify it.
  • Hand-rolled affinity + Kueue ResourceFlavor: keep. It's still the right tool for the placement question it was built for, and nothing above replaces it — KAI Scheduler adds gang scheduling and bin-packing on top, it doesn't take over SKU-level node affinity.

The trigger to actually flip KAI Scheduler on isn't a date — it's the first tenant workload that needs more GPU than one Hetzner node has: a model that doesn't fit in a GEX131's 96GB and has to split tensor-parallel across two boxes. Until that shows up, the fleet's single-GPU-per-node hardware and hand-rolled affinity rules are already fragmentation-proof, in the one sense that matters — there's no multi-GPU node to fragment in the first place.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, including the GPU boxes. Star the repo on GitHub or deploy your first app today.

Sources​

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex
Back to all posts