Your GPUs Are 60% Idle and Kubernetes Says Everything Is Fine

Every pod was running. No crashes, no OOMKills, nothing in the logs. And more than half the GPUs the team was paying for sat idle while training never actually started. That is the opening scene of a CNCF case study published July 23, 2026, in which two Adobe engineers traced ~60% GPU idleness on a Kubeflow cluster to a conflict between the scheduler and the network layer — and fixed it with a few lines of YAML, taking utilization from ~40% to ~85%.

Before the mechanism, the bill. Take one 8-GPU H100 node at 60% idle, priced two ways:

Illustrative numbers, but the shape is the point: on rented hardware an idle GPU burns opex you can cancel; on owned hardware it burns capex you already spent. Idle percentage is the single most expensive metric on a box you own — and Kubernetes' default dashboards cannot see it, because every signal they report was green while the money burned.

This post walks through the case, generalizes it into the three places GPU time actually dies, and gives you the instrumentation ladder to wire up before you conclude you need a second GPU node you may not.

Two correct systems, collectively wrong​

The CNCF writeup's metaphor is worth stealing: a sold-out concert hall, every musician seated with tuned instruments — and the conductor shown to the wrong wing of the building, with the fire-safety doors doing exactly their job by sealing that wing off. Full, expensive hall; complete silence.

The conductor is the training coordinator. The musicians are the GPU workers, which can only play in sync through the coordinator, the way distributed training synchronizes gradients. The fire doors are a Cilium network policy: a correct, intentional zone boundary nobody told the scheduler about. Kubernetes scheduling is topology-agnostic by design — it places pods by resources, not by availability zone — and Kubeflow inherits that assumption. Cilium is topology-aware but cannot reschedule a pod that is already placed. Coordinator lands in one zone, workers in another, and the network silently blocks the connection. No single map shows both the pod placement and the locked doors at once.

What makes the case load-bearing rather than anecdotal is what the authors found next: the same root cause shows up three different ways, on a spectrum from loud to silent:

1. Hard block. A zone-boundary policy denies the connection outright. Training never starts. You notice in seconds — this is the reproduced demo case.
2. Cross-zone latency. No hard block, just distance: gradient synchronization (NCCL AllReduce) pays cross-zone round-trip time on every step, and throughput quietly drops 30–60% with no error at all. You notice in hours, if you are paying attention.
3. Cross-AZ egress. The traffic crosses zones and surfaces only on the cloud bill as inter-AZ transfer. You notice in days, on the invoice.

Symptom 1 fails cleanly. Symptoms 2 and 3 are the ones that quietly drain budgets in real clusters — and on a self-hosted fleet, symptom 3 becomes symptom 2's quieter cousin: no cloud egress line, just permanently degraded throughput on hardware you already bought.

The fix is the punchline of the whole story: nothing about Cilium changed, nothing about Kubernetes or Kubeflow changed. The authors gave the scheduler the one fact it was missing — keep the communicating group together — via nodeAffinity pinning the group to the GPU zone, topologySpreadConstraints co-locating coordinator and workers, and a toleration so the coordinator can land on GPU-tainted nodes. (PodAffinity on the zone topology key does the same thing by relationship, without hard-coding a zone name.) Utilization went from ~40% to ~85% the moment the group was co-located. One fix, all three symptoms.

And the authors' own generalization, quoted because it sets up everything below: any topology-aware CNI plus any distributed ML framework can hit the same wall. It is not specific to Cilium or Kubeflow.

Where GPU time actually dies: three families​

The case covers placement and network. The full picture adds the input pipeline — the failure mode that dominates when there is only one GPU and no network to blame:

• Scheduling and placement gaps. The Kubeflow/Cilium conflict is the vivid instance; the general form is any mismatch between where pods land and where they can efficiently communicate — partial gang scheduling (some workers up, the rest pending, the running ones burning GPU-hours waiting), cross-socket or cross-NVLink-domain placement, one slow rank holding every collective hostage. Signature: high allocation, low occupancy, pods green.
• Network and collective waits. Once placement is sane, multi-node training bottlenecks on gradient synchronization: each GPU waits for the NCCL allreduce to complete before the next forward pass. For mixture-of-experts models, communication overhead can consume up to 43.6% of forward-pass time. Signature: step time dominated by allreduce, SM occupancy collapsing in sync phases, profiling (Nsight Systems NVTX ranges) showing long idle sections inside what looks like compute.
• Input-pipeline starvation. On a single GPU this is the dominant bottleneck: the DataLoader cannot decode, augment, and batch fast enough, so the GPU idles between batches — measurable directly as GPU idle percentage against batch-load time. A July 2026 Nsight profile of a 2-node B300 run found exactly this shape: an NVTX region with nearly every GPU metric dropped to idle inside the forward pass. Signature: low SM activity with low collective time and high data-wait time; the fix is workers, prefetching, and faster storage, not more GPUs.

(There is a minor fourth contributor worth one parenthetical: synchronous checkpointing that blocks the backward pass. Real, but small next to the three above — profile it after the big three are clean.)

Note what is absent from all three families: compute. Nobody in these stories needed a faster card. They needed the cards they had to stop waiting — which is why the industry-wide numbers are so grim. A 2026 optimization-industry survey of tens of thousands of clusters put average GPU utilization across AI/ML workloads at 5%. Your cluster is almost certainly not at 5%. It also almost certainly is not at 85%.

The instrumentation ladder: earn your numbers before you buy hardware​

The CNCF authors' closing advice is the thesis of this section: GPU utilization and pod-zone metrics in Prometheus exposed their failure in seconds; without them it takes days. Here is the ladder, cheapest rung first:

Level 0 — Allocated vs. actual. Join what Kubernetes believes (nvidia.com/gpu requests per pod) with what the silicon reports. If allocated says 8 and the device says 40%, you have the Kubeflow-shaped gap and no dashboard built from pod status will ever show it. This single join is the highest-value query in GPU platform engineering. Alert when the gap persists longer than one training step interval.

Level 1 — DCGM exporter fields. Run dcgm-exporter as a DaemonSet into Prometheus (measured overhead is under 0.5% of GPU throughput at 1-second sampling) and learn three fields: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, and SM occupancy. One critical caveat, documented well by the squire project: GPU_UTIL is a time-based occupancy flag, not a work measurement — it tells you a kernel was resident, not that useful math happened. A card at 90% GPU_UTIL with 15% SM occupancy is stalled, not busy. Alert on SM occupancy below ~20% on an allocated GPU (stalled rank) and on framebuffer pressure above ~95% (OOM approaching, not utilization).

Level 2 — Queue-time vs. run-time, per rank. For scheduled training, compare how long workers wait (pending, rendezvous, allreduce stall) against how long they compute, broken down by rank so stragglers surface individually. A thermal-throttled node or a cross-zone worker shows up here as one rank with systematically worse ratios while its siblings look healthy — the per-rank view is what turns "training is slow" into "replace node 4's cooling" or "worker 7 is across the zone boundary."

Total collection cost for all three levels is on the order of 1.7 MB of compressed time-series per GPU per day. The industry keeps demonstrating the alternative: one practitioner's fine-tuning platform went from 42% to a steady 91% utilization purely by instrumenting first and optimizing what the dashboards showed. Measurement is not overhead here. Measurement is the optimization.

Earn the second node​

The decision rule that falls out of all of this is simple enough to put on a wall: no new GPU hardware until the ladder is green. Allocated-vs-actual gap closed, SM occupancy healthy on allocated cards, per-rank queue-time explained. If those hold and you are still saturated, buy the node with confidence — you have proven demand. If they do not hold, a second node just doubles the waste: the same starving DataLoader, the same cross-zone collectives, now spread over twice the capex.

This is also where the owned-vs-rented framing cuts both ways, and honesty requires saying so. Falling H100 rental rates (~$2–3/hr on specialized clouds in 2026, down roughly two-thirds from what teams budgeted two years ago) make renting the rational way to absorb bursts and to validate demand before a hardware purchase. Ownership wins when utilization is high and sustained — which is exactly what the ladder produces. The sequence is: rent to discover the workload, instrument to fix the waste, own to harvest the margin. Buying the node first and profiling later is how teams end up owning very expensive space heaters.

Your scheduler cannot see your network topology. Your pod status cannot see your silicon. The gap between those two blindnesses is where GPU budgets go to die — roughly 60% of them, in the best-documented case we have. Close the gap with a join, three DCGM fields, and per-rank queue-time before you reach for the purchase order.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. The same Cluster-API fleet that serves your apps can schedule your agent sandboxes and inference workloads next to your training jobs, with the utilization dashboards above telling you when the next GPU node is actually earned. Star the repo on GitHub or deploy your first app today.