Hot-Pluggable GPUs Join the CNCF: What CoHDI's Sandbox Debut Means for Your Self-Hosted GPU Pool

Your GPU nodes have a property that your pods don't: their hardware is frozen at boot time. A pod can request a GPU, release it, and let another pod claim it — but the GPU itself stays bolted to the node it shipped with. If the node with the free GPU is running prefill traffic while the node starving for decode capacity sits next door, the scheduler can't help you. That is a hardware problem wearing a scheduling costume.

On July 28, 2026, a new CNCF Sandbox project said it wants to take the costume off. CoHDI (Composable Hardware in Disaggregated Infrastructure, pronounced "Cody") promises host-level attach and detach of PCIe devices — GPUs included — on running nodes, without reboots, orchestrated through Kubernetes' Dynamic Resource Allocation (DRA). The headline use case is disaggregated LLM inference: prefill and decode split across separate pools with different hardware needs.

Here is the verdict up front, because the rest of this post is evidence for it: CoHDI is a watch, not an adopt. The idea attacks a real cost in small self-hosted GPU pools, and the DRA foundation it builds on is genuinely production-grade. But Sandbox means experimental, the hardware fabric it needs is absent from typical rented bare metal, and everything it promises for your two-GPU fleet can be approximated today with DRA on fixed GPUs. This post shows the before/after math, the Cluster API implications, and the exact graduation signals to watch.

What CoHDI actually is: three components between your pods and the PCIe bus​

CoHDI launched in March 2025 as a collaboration between Red Hat, FSAS Technology, Fujitsu, IBM Research, and NTT (originally under the name InfraDDS), and entered the CNCF Sandbox in July 2026. Its scope is deliberately narrow: make individual PCIe devices — GPUs, and eventually other accelerators and NICs — attachable to and detachable from a running Kubernetes node, driven by pod demand, with no OS reboot in the path.

The software suite has three cooperating pieces:

The key architectural choice is that CoHDI does not invent its own scheduling API. It rides on DRA — the structured ResourceClaim / ResourceClass / DeviceClass model that reached general availability in Kubernetes v1.35 and is now, per the CNCF's July 2026 explainer, the standard way to express "this pod needs that device" with topology, sharing, and driver-version semantics the old nvidia.com/gpu: 1 extended-resource hack could never carry. NVIDIA's donation of its DRA GPU driver to the CNCF at KubeCon Europe 2026 moved the reference implementation under community governance, which is the single strongest signal that DRA is the stable floor CoHDI stands on.

So the dependency stack reads bottom-up: composable PCIe fabric (rare, expensive, vendor-specific) → CoHDI manager + operator + scaler (new, Sandbox) → DRA (GA, boring in the good way). When this post says "watch, don't adopt," the top two layers are the reason and the bottom layer is the consolation prize: you can bank the bottom layer today.

Why inference disaggregation is the workload that justifies it​

To understand why anyone would rebuild host hardware topology for this, you need the two phases of LLM inference, because they stress GPUs in almost opposite ways:

• Prefill processes the whole prompt at once. It is compute-bound: FLOPS-hungry, bursty, latency-sensitive in the time-to-first-token (TTFT) dimension. A traffic spike means a sudden need for lots of compute, right now.
• Decode generates tokens one at a time, each step reading the accumulated KV cache. It is memory-bandwidth-and-capacity-bound: what matters is VRAM size and bandwidth per concurrent sequence, and the metric to protect is time-per-output-token (TPOT).

Running both phases on the same GPU pool forces an unhappy compromise: provision for the prefill bursts and your decode capacity idles between them; provision for steady decode and your prefill queue explodes TTFT under load. Prefill-decode (PD) disaggregation, now the standard architecture for LLM serving, splits the two phases onto separately sized pools connected by a KV-cache transfer, so each pool can be optimized — and scaled — for its own bottleneck.

The Kubernetes-native shape of this is already visible without any composable hardware. The llm-d project runs prefill and decode workers in a single InferencePool, distinguished by role labels, with the endpoint picker routing each request phase to the right role. That is software disaggregation on fixed GPUs: the pools are logical, and moving capacity from decode to prefill means rescheduling pods, not rewiring hardware. CoHDI's pitch is to make the pools physical too — detach a GPU from a decode-heavy node and attach it to the prefill node for the duration of the spike, then hand it back.

For a hyperscaler running thousands of inference GPUs, that physical flexibility is obviously valuable. The question this post exists to answer is whether any of it survives contact with a fleet of two.

The worked example: two owned GPUs, one traffic spike​

Fix the fleet so the math is honest. Assume a small self-hosted setup: two rented bare-metal GPU nodes (Hetzner GEX44-class — one datacenter GPU each, no composable PCIe fabric, no CXL memory pooling, plain PCIe slots on commodity boards), running one inference service with disaggregated prefill and decode roles on fixed GPUs, plus ordinary tenant web workloads sharing the same nodes.

Before (today, fixed GPUs). Each GPU is a node property. The scheduler can move pods between nodes, but it cannot move a GPU to the load. A representative evening looks like this:

The waste is structural, not operational. GPU B has idle compute during the burst and GPU A has queued demand, and no pod reschedule fixes it because the constraint is which board the silicon is soldered next to. Operators compensate the usual ways: overprovision a third GPU (capex you can't justify at this scale), cap concurrency (throughput you can't afford to lose), or let TTFT degrade (the SLO you promised). At two GPUs, every one of those hurts proportionally more than at two thousand.

After (the CoHDI promise, on hypothetical fabric). The burst arrives; the prefill ResourceClaim can't be satisfied on GPU A; the Composable-DRA-Driver surfaces GPU B's device as claimable; the operator detaches it from the decode node and the scaler attaches it to the prefill node — all without reboots — for the burst duration, then reverses when decode pressure returns. Utilization of the fleet's scarcest resource (compute during bursts, VRAM during steady decode) tracks demand instead of topology.

Now the limits box, which is the actual point of this section:

• The fabric doesn't exist in this fleet. Composable PCIe attachment needs fabric hardware (PCIe switching, CXL-class interconnects, vendor manager APIs) that rented Hetzner-style bare metal simply doesn't expose. On today's fleet the "after" column is a thought experiment, not a migration plan. Label it accordingly.
• Attach latency vs. burst timescale. A prefill burst needs relief in seconds; a device detach, fabric re-route, driver bind, and kubelet device re-discovery pipeline measures in tens of seconds at best, minutes realistically in v1 Sandbox code. If your bursts are shorter than the attach loop, CoHDI adds churn without cutting TTFT.
• Two new controllers in the failure path. The scaler and the operator sit between "pod needs GPU" and "pod gets GPU." Their bugs — partial detach, device stuck claimed, node left with a phantom ResourceSlice — are new failure modes with no runbook yet, on the exact path your inference SLO depends on.
• At two GPUs, the granularity is the fleet. Detaching GPU B to save prefill starves decode entirely; there is no third pool to absorb the shift. Fractional mechanisms (MIG slices, time-slicing, HAMi-style pooling — see below) move capacity in increments smaller than a whole card, which is what a two-GPU fleet actually needs.

Read the example as: the pain is real and correctly diagnosed, but the prescribed hardware cure doesn't fit the patient's building.

What this would mean for Cluster API operations​

The TODO behind this post names a Cluster-API-managed fleet explicitly, so here is the ops mapping instead of hand-waving.

Today, a GPU node pool is a MachineDeployment (or equivalent) with a GPU machine type, and GPU-ness reaches the scheduler through node labels and affinity: "this pod needs accelerator=nvidia-l4, schedule it on nodes carrying that label." Cluster API reconciles machines; the device plugin advertises the GPU; DRA — if you've adopted it — replaces the label haggling with ResourceClaims against fixed devices. The failure domain is clean: a machine is a machine, its GPU is part of it, and MachineHealthChecks reason about both together.

Under CoHDI, that coupling breaks on purpose. The GPU becomes a scheduled resource with its own lifecycle, and two controllers now mutate what "this machine" means:

• DRA decides which device satisfies a claim, independent of node.
• The CoHDI operator + scaler change which node hosts the device, live.

Cluster API must then tolerate host device churn it never signed up for: a healthy machine whose GPU just detached looks, to today's health checks and autoscalers, like a machine that lost its reason to exist — or worse, like a failure worth remediating by replacing the node. Until CAPI, the autoscaler, and CoHDI agree on who owns "machine with N attached devices" as a reconciled object (rather than three controllers tugging at it), the ops story is "run it in a lab cluster and watch the controllers argue," not "roll it to the tenant pool."

That is also why the software-only path in the next section is operationally cheaper: it keeps exactly one owner of machine identity.

The maturity verdict: what "ready" would have to look like​

DRA GA is the baseline to measure CoHDI against, because it's the part of the stack you can already bet tenant traffic on: GA API in v1.35, CEL-selected DeviceClass, vendor drivers (NVIDIA's now community-governed, AMD and Intel shipping their own), and a migration story from extended resources via KEP-5004's synthesized claims. If your fleet isn't scheduling fixed GPUs through DRA claims yet, that migration — not CoHDI — is the highest-return GPU-scheduling work available to you.

Against that bar, CoHDI's Sandbox checklist writes itself:

Exit criteria before a tenant-facing bet: Incubation graduation (or at minimum multi-release API stability), at least two independent fabric vendors with documented attach-latency SLOs, public production references running inference disaggregation on it, and a written Cluster API coexistence story for device churn. A realistic place to check for those signals is KubeCon + CloudNativeCon North America in November 2026, where Sandbox projects either show adoption momentum or don't.

What to do now instead, in priority order: schedule fixed GPUs through DRA ResourceClaims so your manifests are CoHDI-shaped when the hardware ever arrives; run prefill/decode as separate llm-d roles on fixed GPUs to capture most of the disaggregation win in software; and share whole cards across tenants with MIG partitions, time-slicing, or HAMi-style heterogeneous accelerator pooling before buying more silicon. None of these need a fabric, all of them shrink the exact waste the worked example measured, and the first one is forward-compatible with everything CoHDI might become.

CoHDI's bet — that the GPU stops being a node property and starts being a scheduled, hot-pluggable resource — points in the right direction. At Sandbox stage, the right move for a small self-hosted fleet is to face that direction while standing on the GA ground: DRA claims today, disaggregated roles on fixed GPUs today, composable fabric when someone can sell it to you with an SLO.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. If disaggregated inference on owned hardware is on your roadmap, start where this post ends: DRA claims and prefill/decode roles on the GPUs you already rent. Star the repo on GitHub or deploy your first app today.