One GPU, Two Tenants: Where DRA, MIG, vCluster, and DCGM Share a Card — and Where the Trust Boundary Says Stop

CNCF's August 2026 AI-factory guide puts the whole GPU-sharing stack on one page: Cluster API and GitOps for machine lifecycle, DRA plus MIG and HAMi for allocation, vCluster for tenant control planes, and DCGM with OpenCost for utilization and chargeback. It reads like a shopping list. The question it leaves to the operator is the only one that matters: which of these layers actually isolate tenants from each other, and which ones just schedule them?

Here is the verdict up front, for the concrete scenario — two tenants, one H100-class card on a self-hosted box, Tenant A running your own team's agent sandboxes, Tenant B running an external customer's inference endpoint:

The one-sentence rule: share silicon only within one trust domain, share control planes only at vCluster-or-better, and bill on allocation while auditing with utilization. Everything below is why each half of that sentence holds.

The tempting demo and what it hides​

"One card, two customers" demos beautifully. Two pods, one nvidia.com/gpu each — or a MIG profile split down the middle — both green in kubectl get pods, utilization graphs that finally look respectable instead of a single tenant idling at 30%. Red Hat's multitenant GPU infrastructure writeup names the trap precisely: most production failures in shared GPU environments are not hardware limitations but misalignment between isolation layers. The GPU is partitioned while the scheduler, the runtime, and the network assume it is not — or the reverse.

So walk the stack bottom-up, and grade each layer on the only axis that counts: what happens when Tenant B is not merely noisy but adversarial.

Layer 1: splitting the silicon — MIG vs time-slicing vs HAMi​

Three ways to cut one card, with very different guarantees:

• MIG (Multi-Instance GPU) is the only option here that is hardware. Each instance gets dedicated memory controllers and compute engines, so cross-instance interference is physically impossible and an OOM in one slice is strictly contained. This is the correct answer for "two tenants, one card" — with four honest limits. It needs Ampere-class or newer silicon. Each node runs a single MIG strategy, so slice sizes are a node-level decision, not a per-pod one. NVLink does not span slices, which kills tensor-parallel multi-GPU inference across them. And any model that needs more than ~40GB simply does not fit — it wants the whole card or several.
• Time-slicing / MPS advertises one card as N integer replicas. It improves utilization and costs nothing, but the replicas share the GPU with no memory or fault isolation and no proportional-compute guarantee. Fine for your own team's batch jobs. Not a tenant boundary.
• HAMi sits in the middle as middleware: virtual-GPU slicing with memory limits and compute throttling enforced in software. Better accounting than time-slicing, weaker guarantees than MIG silicon. As DRA matures past device counting, the question is whether HAMi remains a layer or becomes a migration source — more on that next.

The verdict for this layer: MIG within one trust domain, whole cards across trust domains, time-slicing only where you would already share a process boundary.

Layer 2: telling the scheduler — DRA grows up​

The legacy device-plugin model counts whole GPUs as integers and knows nothing about slices, sharing modes, or topology. Dynamic Resource Allocation replaces counting with claiming: a ResourceClaim describes what the workload needs, the DRA driver publishes per-node state (GPU model, MIG profiles, consumable capacity), and the scheduler matches them. The timeline matters for planning: alpha in 1.26, beta in 1.30, stable upstream — with the NVIDIA DRA driver now supporting MIG devices, consumable-shares accounting, and per-claim sharing configuration (time-slicing vs MPS vs exclusive) attached directly to the claim instead of a node-wide device-plugin flag.

For the two-tenant design, DRA buys two concrete things the device plugin cannot. First, heterogeneous sharing on one node: Tenant A's sandbox claims a MIG slice while Tenant B's job claims a whole GPU, expressed in the same API instead of two device-plugin configurations fighting over one kubelet. Second, fault isolation hooks: DRA device taints let an operator fence a single degraded GPU without cordoning the node, so one bad card stops taking every other tenant's job down with it.

The migration caveat is real: workloads that request nvidia.com/gpu keep working, but nothing gets DRA's benefits until claims are written, drivers are deployed per node pool, and autoscaler behavior against claims is verified. Budget that work per pool, not per cluster.

Layer 3: separating the control planes — what vCluster is and is not​

Namespaces are not a tenant boundary for GPU workloads: quotas do not cover device memory semantics, and every tenant shares one API server, one scheduler, and one blast radius for RBAC mistakes. The vCluster argument — made at length by its maintainers — is that virtual clusters give each tenant its own control plane (its own API server, its own scheduler, even its own Kai scheduler or policy engine) while pods land on shared host nodes for utilization.

Grade it honestly. vCluster isolates the Kubernetes control plane: Tenant A cannot list Tenant B's objects, break Tenant B's scheduling, or consume Tenant B's API budget. It does not isolate the host: CRI, CNI, kubelet, kernel, and GPU driver are shared. That is exactly the right boundary for two teams inside one company — and exactly the wrong one to sell as isolation to two mutually untrusted customers. For that, the same vendor's answer is dedicated worker nodes per tenant ("Private Nodes") or virtualized nodes (vNode) for harder separation: keep the vCluster control-plane split, but stop sharing the kernel.

Map it to the scenario: Tenant A (your agent sandboxes) and your other internal teams share nodes under separate vClusters. Tenant B (external inference) gets its vCluster scheduled onto dedicated nodes, or whole-GPU claims fenced by taints — never a MIG slice adjacent to an internal tenant on a shared host if the contract promises isolation.

Layer 4: making the bill honest — DCGM plus OpenCost​

Sharing without accounting is subsidizing. Two metrics systems do two different jobs:

• OpenCost answers allocation: which namespace, pod, and tenant held which GPU slice for how long, at the rate-card price of a GPU-hour. This is what you invoice — it is deterministic and auditable from Kubernetes state.
• DCGM exporter (shipped with the NVIDIA GPU Operator) answers utilization: DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_DEV_FB_USED — what fraction of the streaming multiprocessors did work, how much VRAM was actually touched. This is what you audit with.

The FinOps-for-AI pattern that keeps showing up is the combination: attributable cost ≈ (GPU memory consumed by pod ÷ total GPU memory) × hourly node cost, with DCGM utilization exposing the gap between allocated and used. That gap is the platform decision. Tenant A holding a MIG slice at 25% SM activity is your own team's utilization problem to fix with packing. Tenant B holding a whole GPU at 25% is revenue — provided the rate card priced the whole card, not the FLOPs.

One accounting trap to avoid: time-sliced replicas look like N GPUs in allocation data while being one GPU in utilization data. If any pool still runs time-slicing, reconcile the two sources before invoicing, or Tenant B pays for a card it shared with three neighbors.

The worked design: one box, two tenants​

Concretely, on a single 8-GPU node managed by Cluster API with a GitOps lifecycle:

1. Partition two cards with MIG (say, 1g/2g profiles) for internal density; leave the rest whole for large models and external tenants. One MIG strategy per node — decide at provisioning time, in git.
2. Serve slices and whole GPUs through DRA claims, not extended resources, so the scheduler sees both shapes and a degraded card can be tainted individually.
3. One vCluster per tenant. Internal tenants share the MIG pool; the external tenant's vCluster is constrained (node selector + taint toleration) to whole-GPU nodes.
4. DCGM + OpenCost per namespace, one dashboard for allocation, one for utilization, reviewed together monthly — the pair is the chargeback system, neither alone suffices.
5. Write down the refusal list: no time-sliced sharing across trust domains, no MIG adjacency for tenants promised isolation, no external workload on a node without dedicated-node scheduling. The boundary is a document, not a vibe.

When the trust boundary says they cannot​

Three cases where the answer is a second card, a second node, or a second cluster — no cleverness available:

• Regulated or contractual isolation. If the contract says "dedicated," a MIG slice on a shared kernel is a breach with extra steps. Whole GPU minimum, dedicated node preferred.
• Mutually untrusted code execution. Agent sandboxes running arbitrary generated code next to a customer's inference weights is the threat model that ends sharing. The shared kubelet, driver, and host kernel are the attack surface no scheduler feature removes.
• Performance SLOs with teeth. MIG removes compute/memory interference but shares power, thermals, PCIe bandwidth, and NVLink fabric behavior. A latency SLO measured in tokens per second under a noisy neighbor belongs on a whole card.

None of this argues against sharing — an internal fleet at 30% utilization is money on fire, and MIG plus DRA plus vCluster is the best version of density Kubernetes has ever had. It argues for drawing the line where the failure mode changes from "our team's job queued" to "our customer's data, SLO, or contract broke." That line runs between Tenant A and Tenant B, straight through the middle of the card.

Need somewhere to run this? Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.