KEP-5677: Tell Tenants 'No GPU Left' Up Front Instead of Parking Their Pods in Pending

Your pod is Pending. Your bill is ticking. You deployed a GPU workload, the scheduler replied with the line every platform team has memorized — 0/N nodes available: Insufficient nvidia.com/gpu — and now your pod sits in the queue telling you nothing about what is actually free, where, or for how long. On a hyperscaler you throw money at the opacity: add another node pool, over-provision, move on. On a self-hosted fleet with a handful of scarce GPU machines, that opacity is the whole capacity-planning conversation, and you are losing it.

The short version: Kubernetes 1.37's second alpha of KEP-5677, announced in the September 3 DRA updates post, finally answers "what GPU capacity is free?" with a real API object. You create a ResourcePoolStatusRequest, a controller computes a per-pool available/allocated snapshot exactly once, and you read it back. A self-hosted PaaS can turn that snapshot into two things tenants have never had: a capacity page showing free GPUs per pool, and a fast "no GPU right now, retry or queue" at admission time instead of a pod parked in Pending for hours. Here is what shipped, what is deliberately missing, and the concrete pattern to build on it.

What actually shipped in 1.37: availability visibility, second alpha​

Dynamic Resource Allocation knows everything about your accelerators and tells you almost nothing. ResourceSlice objects publish total device capacity per pool; ResourceClaim objects track individual allocations. But nothing in the API subtracts one from the other. A developer whose pod fails to schedule cannot see available versus consumed capacity; a cluster admin planning expansion has to stitch together cluster-scoped slices with namespaced claims across every namespace — claims they may not even have RBAC to read. The KEP's motivation section names the three wounds plainly: debugging difficulty, no capacity planning primitive, and no cross-namespace consumption view.

ResourcePoolStatusRequest closes that gap with a deliberately boring mechanism: the CertificateSigningRequest pattern. Create a request object naming a driver, wait for the controller to compute, read the answer from .status:

apiVersion: resource.k8s.io/v1alpha3
kind: ResourcePoolStatusRequest
metadata:
  name: check-gpus-1757960000
spec:
  driver: gpu.example.com   # required: bounds the answer to one driver's pools
  # poolName: hetzner-gpu-01  # optional: narrow to a single pool

kubectl wait --for=condition=Complete \
  resourcepoolstatusrequest/check-gpus-1757960000 --timeout=30s
kubectl get resourcepoolstatusrequest/check-gpus-1757960000 -o yaml

And the status comes back as plain counts per pool:

status:
  poolCount: 2
  pools:
  - driver: gpu.example.com
    poolName: hetzner-gpu-01
    nodeName: hetzner-gpu-01
    generation: 3
    resourceSliceCount: 1
    totalDevices: 4
    allocatedDevices: 4
    availableDevices: 0
    unavailableDevices: 0
  - driver: gpu.example.com
    poolName: hetzner-gpu-02
    nodeName: hetzner-gpu-02
    generation: 3
    resourceSliceCount: 1
    totalDevices: 4
    allocatedDevices: 1
    availableDevices: 3
    unavailableDevices: 0
  conditions:
  - type: Complete
    status: "True"
    reason: CalculationComplete

That is the whole primitive: total, allocated, available, unavailable — per pool, per node, one driver at a time. It landed as a first alpha in 1.36 and moved to a second alpha in 1.37, behind the DRAResourcePoolStatus feature gate (default off), which must be enabled on both the API server and kube-controller-manager.

Two operating rules define everything you build on top. First, status is write-once: once the controller sets it, the object is immutable, and there is no update-in-place. To refresh, you delete and recreate the request. Second, stale requests clean themselves up: completed or failed requests are deleted one hour after their condition is set, and requests that never get processed (controller down, gate mismatch) are swept after 24 hours. The KEP's own starter pattern for automation is a cron loop that does exactly this every five minutes — create, wait, read availableDevices, alert if low, delete. That loop is the honest operating cost of the API: polling you own, on a cadence you choose.

What's in the snapshot — and what's deliberately not​

The request spec is three fields. driver is required and must be a DNS subdomain; it is what bounds the response so one request cannot dump the whole cluster. poolName optionally narrows to a single pool. limit caps the pools returned — default 100, max 1000, with the status list hard-capped at the same ceiling. Each returned pool carries its generation (the highest slice generation observed; older generations are silently ignored, not counted as errors), the slice count, the four device counters, and — when the driver's published slices are fewer than the driver declared — a validationError with the counts left unset while the controller requeues up to five times to let the driver catch up.

The 1.37 round of the alpha fixed the two ways 1.36's counting lied on modern hardware. Partitionable devices (one physical GPU exposed as mutually exclusive partitions sharing a counter set) and consumable devices (one device serving many claims at once) both broke naive device counting — the latter could drive availableDevices to zero on pools that still had free capacity. Alpha 2 adds a partitionSummary view and a shareableSummary aggregate per pool, caps each device's contribution to allocatedDevices at one, and skips AdminAccess allocations in all accounting. It also made unavailableDevices real: in 1.36 it was always zero, while 1.37 derives it from actual device taints with NoSchedule and NoExecute effects — the same taints whose device-level support went stable in 1.37, so taking one GPU offline for maintenance now shows up in the snapshot instead of silently shrinking the schedulable pool.

What the API refuses to be matters more than what it is. The KEP's non-goals are explicit: no real-time metrics, no quotas or limits, no historical consumption data, no watch support. And the snapshot exposes counts only — pool names, driver names, node names, device numbers. It never reveals which claims hold which devices, any claim contents or pod information, raw slice data, or cross-namespace claim details. A tenant holding the read role learns "pool B has 3 of 4 free" and nothing about whose workloads hold the other five. That information boundary is what makes the next section safe to build: RBAC grants are explicit (no new default roles, not aggregated to admin), and even a broad grant leaks only arithmetic.

The PaaS play: a capacity page and a fast "no" at admission time​

Take the concrete fleet this primitive was made for: a self-hosted platform on two scarce Hetzner GPU machines, running tenant AI-agent sandboxes — inference-backed, bursty, and allergic to waiting. Today a tenant submits a sandbox deploy when both pools are full, and the platform's answer is silence: a pod in Pending against an opaque scheduler queue, discovered whenever the tenant thinks to run kubectl describe. The snapshot turns that silence into two artifacts you can ship this quarter.

Artifact one: a capacity page. A small platform controller creates one ResourcePoolStatusRequest per GPU driver on a short cadence — every minute, not the KEP's illustrative five, is cheap here since each request is one controller computation over informers KCM already holds — and renders the cached result as per-pool available/total counts stamped with the Complete condition's lastTransitionTime:

The timestamp is not decoration; it is the freshness contract. Tenants learn to read "3 free as of 40 seconds ago" the way they read any cached dashboard, and the platform never pretends a point-in-time read is a live feed. Because the underlying API exposes counts only, this page can be tenant-visible without a per-tenant filtering layer — there is nothing in the payload to filter.

Artifact two: the admission-time fast "no." The deploy pipeline consults the same cached snapshot before submitting a GPU-backed claim. If no pool shows a free device matching the request, the tenant gets an answer in seconds — "no GPU capacity right now; retry, queue, or pick a CPU sandbox" — instead of a pod that sits Pending for hours and fails the same way at 3 a.m. that it would have failed at submit time. Side by side, the UX change is stark:

Note what this is not: it is not a reservation system, and the snapshot's point-in-time nature means two tenants can both see "1 free" and race for it. The admission check is advisory — the scheduler remains the authority, and the check's job is to convert the certain failure (zero free anywhere) into a fast answer while letting the contended case proceed to normal scheduling. For a scarce pool where "full" is the common state, filtering the certain failures alone clears most of the queue.

What still needs real monitoring (and the alpha fine print)​

A snapshot you poll is not observability, and the KEP is admirably honest about the gap. Before depending on this in production, walk the checklist:

• It is alpha, twice over. The gate is off by default and must be flipped on both the API server and kube-controller-manager; a gate mismatch leaves requests forever without status (detectable — no Complete or Failed condition — and swept after 24 hours). Per-user rate limiting for request creation is explicitly deferred to Beta, so your controller's cadence plus the built-in TTLs (1 hour after completion, 24 hours if pending) and work-queue rate limiting are the current flood protection. If tenants can create requests directly, put an admission policy (Gatekeeper, Kyverno) bounding object counts in front of them.
• Staleness is yours to manage. Every snapshot carries its age in lastTransitionTime; nothing pushes updates. Size your refresh cadence to your scheduling churn — a minute for a busy sandbox fleet, five for a quiet one — and surface the timestamp everywhere the numbers appear.
• Driver health and real utilization live elsewhere. The snapshot counts allocations, not health or load. A wedged DRA driver, a GPU running at 100% VRAM with one allocated claim, a node whose kubelet stopped preparing claims — none of that appears in pool counts. Prometheus driver metrics, device-health signals, and claim-level status remain the monitoring underneath; the snapshot answers "is there room," never "is it working."
• RBAC is explicit by design. No default roles include the new API, and the KEP keeps it out of the admin aggregation. Grant create/get/list/delete on resourcepoolstatusrequests deliberately — your platform controller needs it; most tenants only need the capacity page and the admission answer your pipeline derives from it.

One more honest limit: the incomplete-pool path. When a driver's published slices lag its declared count, the pool reports validationError with counts unset, and the controller retries five times. Your capacity page should render that state as "unknown," not zero — conflating "driver still publishing" with "pool full" would turn a transient publish lag into phantom rejections at admission time.

Why a snapshot is the right shape for an admission decision​

It is tempting to read "no watch support" as a missing feature. It is the design. An admission decision — admit or refuse this deploy — must be stable for the duration of the decision. A streaming availability feed would flap under you: free at evaluation, gone at submit, free again before the error page renders, with every flap an argument between your pipeline and the scheduler about what "available" meant at which instant. A timestamped point-in-time read sidesteps the race by being explicit about what it is: this was true 40 seconds ago; act accordingly. The scheduler stays the single authority on placement; the snapshot is an advisory input with a known age, and advisory inputs with known ages compose cleanly with authoritative systems.

That is also why the natural Beta/GA progression — per-user rate limits, possibly configurable TTLs — extends the primitive without changing its shape. The API that refuses to be a stream will still refuse, correctly, when it graduates.

For the self-hosted fleet, the timing compounds. In the same 1.37 release, DRA Extended Resource support went GA, so existing workloads written against example.com/gpu-style extended resources keep working unmodified while allocation logic moves to DRA underneath — the gradual-migration story. KEP-5677 is the other half of that migration's operability: once your GPUs are DRA-managed, you can finally see them. A capacity page and a fast "no" are not glamorous features. They are the difference between a platform whose tenants trust its GPU tier and one whose tenants learn that submitting means waiting, and waiting means nothing.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.