One Bad GPU Shouldn't Sink the Node: Kubernetes 1.37's DeviceTaintRule Goes Stable

One of the eight GPUs on your node starts throwing Xid errors at 2 AM. Until this month, your only upstream move was kubectl cordon: fence the entire node, strand the seven healthy cards alongside the sick one, and reschedule every tenant workload that happened to live there — including the ones nowhere near the broken hardware. That is 7/8ths of a node's GPU capacity sacrificed to isolate one card, and on hardware you own outright, stranded capacity is money burned with nothing to show for it.

Kubernetes 1.37 fixes the granularity. The fix, up front, is a single object:

apiVersion: resource.k8s.io/v1
kind: DeviceTaintRule
metadata:
  name: gpu-node-3-card-5-unhealthy
spec:
  deviceSelector:
    driver: gpu.example.com
    pool: worker-gpu-03
    device: gpu-5
  taint:
    key: gpu.example.com/unhealthy
    value: Xid-79
    effect: NoExecute

Apply that and the scheduler stops placing new pods on gpu-5, while the device-taint eviction controller removes the pods already using it — and the other seven cards on worker-gpu-03 keep serving tenants untouched. The rest of this post is the three things that YAML assumes: what each effect actually does, how to roll it out without evicting the wrong tenants, and what your health-check pipeline has to build so a human is not hand-writing that object at 2 AM.

The effects mirror node taints exactly, at device granularity:

Three selector details decide whether the rule does what you meant. First, one DeviceTaintRule adds exactly one taint — fencing two bad cards takes two rules. Second, a rule with no selector matches nothing, deliberately: upstream made the empty selector a no-op so a forgotten field cannot evict every pod holding a ResourceClaim. Third, you scope down driver → pool → device, and for node-local devices the driver is encouraged to use the node name as the pool name — which means pool-only selection is also how you taint every device on a node when the fault really is node-wide.

One sharper edge: drivers may publish stable names like gpu-0 that hide which physical card sits behind the name, so rules support CEL selectors matching a vendor-specific unique-ID attribute when the driver exposes one. Check the driver's docs for real names before copying the example above; the shape is stable, the strings are driver-specific.

Why you can rely on it now: stable in 1.37, nothing to enable​

This is KEP-5055: driver-published taints arrived as alpha in v1.33, the admin-authored DeviceTaintRule followed in v1.35, the pair went beta in v1.36, and 1.37 promotes both to stable with the DRADeviceTaints and DRADeviceTaintRules feature gates locked — set them explicitly and the API server ignores the value without even an error.

The API lives at resource.k8s.io/v1, and the v1.37 DRA updates post states the contract plainly: drivers can mark devices tainted so they are skipped for new pod scheduling, admins can apply the same taints cluster-wide via a DeviceTaintRule without reconfiguring drivers, and pods already using a tainted device can be evicted automatically unless their ResourceClaim explicitly tolerates the taint. Stable here means the blast-radius primitive is now as load-bearing as a node taint — safe to build fleet automation on top of.

The safe rollout: dry-run with None before you evict with NoExecute​

Going straight to NoExecute on a shared multi-tenant node is how you turn one bad card into a self-inflicted multi-tenant eviction. The upstream docs describe a dry-run flow built on the None effect, and it is worth running verbatim:

1. Apply the rule with effect: None. The scheduler and eviction controller ignore it, but the eviction controller still computes who would be affected and writes it into the rule's status.
2. Read the status message. It looks like this: "3 published devices selected. 1 allocated device selected. 1 pod would be evicted in 1 namespace if the effect was NoExecute." Published devices are those listed in ResourceSlices — tainting them blocks new allocation; only allocated devices cause evictions. Note the caveat: the message is computed once, not refreshed — recreate the rule to recompute.
3. Flip the effect to NoExecute and wait. kubectl wait --for=condition=EvictionInProgress=false DeviceTaintRule/<name> blocks until the eviction controller reports the wave done.

Two shock absorbers come free with the model. ResourceClaims can tolerate taints — an empty toleration matches all of them — so a tenant that genuinely must ride out degraded hardware (a checkpoint-draining batch job, say) can opt in explicitly rather than being protected by accident. And eviction can be delayed with tolerationSeconds, counted from the taint's timeAdded stamp, which the API server sets automatically and updates whenever the effect changes. The documented race — scheduler and eviction controller observing a new taint at slightly different times, so a pod lands just as the controller declares evictions finished — is damped by an intentional few-seconds delay before the status condition flips. Unlikely is not impossible, so treat the wait as a gate for automation, not a proof for auditors.

What your health-check pipeline has to build​

A DeviceTaintRule is a fence, not a smoke detector. Upstream split the responsibility on purpose: drivers report device state, operators decide what is actionable. Concretely, the chain from a dying card to an authored rule has three links, and today you own all of them:

• Detect. The signals are the ones GPU operators already collect: DCGM_FI_DEV_XID_ERRORS nonzero, ECC double-bit errors climbing, dcgmi diag failures, or Xid signatures (79, 74, 92, 95 are the classic fatal set) in the kernel log. The established pattern is a Node Problem Detector custom plugin or a DCGM-exporter alert that turns those signals into something the control plane can see.
• Decide. A controller — yours, or driver health-checking as it matures — maps the signal to a taint key and effect. Degraded-but-running (rising thermals, correctable ECC slope) gets NoSchedule: drain by attrition, no evictions. Dead or faulting (Xid 79, GPU lost) gets the None dry-run first, then NoExecute.
• Author. The controller creates the DeviceTaintRule scoped to driver + pool + device, or to the vendor unique ID via CEL when the driver's published names are unstable across restarts.

The honest gap sits in the middle link. The NVIDIA DRA driver's health story is still catching up to the API: v0.4.0 of the driver wires an NVMLDeviceHealthCheck behavior change to KEP-5055, and the driver's own tracking issue scopes the remaining work as defining a taint schema for NVML events (GPU lost, ECC errors) that distinguishes fatal from non-fatal Xids before promoting the check toward GA. Until that lands, driver-published health taints are the exception and operator-authored rules are the norm — which is exactly why the admin-side DeviceTaintRule graduating to stable matters more than it looks: the half of the feature you can use today, without waiting on any driver's health roadmap, is the half that just went GA. In the legacy device-plugin world there is no equivalent at all: hyperscaler guidance to this day phrases GPU fault response as "Xid errors → taint the node," the whole node, all its tenants, every time.

Three limits to write on the whiteboard before you adopt it​

• DRA-published devices only. Taints attach to devices a DRA driver publishes in ResourceSlices. A fleet still on the classic device-plugin path gets nothing from this feature — its migration to DRA claims is the actual prerequisite, and that migration (slicing model, claim templates, scheduler integration) is the bigger project.
• The control plane must be 1.37. Scheduler, API server, and controller-manager all participate — scheduling, taint bookkeeping, and the eviction controller respectively. A 1.36 cluster with the beta gate on gets the behavior, but mixed-version skew during the upgrade is exactly when the scheduler/eviction race is likeliest, so roll the control plane first and fence cards after.
• One taint per rule, one rule lifecycle per incident. The taint lives exactly as long as the rule object. That makes GitOps natural (a rule per incident, reverted on repair) and makes stale rules a real failure mode: a forgotten NoSchedule on a replaced card silently shrinks your schedulable fleet. Expire or reconcile rules from the same pipeline that authors them.

The blast radius, per tenant​

Step back to the 2 AM page. Before 1.37, one bad card on a shared node meant cordoning the node: every tenant on that box reschedules, healthy cards idle, and the platform eats the capacity loss plus the eviction storm. After 1.37, on a DRA-based fleet, one bad card means one DeviceTaintRule: the affected tenant's pods on that card evict (or tolerate, if they opted in), every other tenant on the node never notices, and seven cards keep earning their keep. That is the whole argument — fault isolation at the granularity of the failing hardware, as a stable upstream API instead of a platform-specific hack. Build the detector once, and every GPU node in the fleet gets per-card fencing for free.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Sources​

• Kubernetes v1.37: DRA Updates — kubernetes.io, September 2026: device taints and tolerations stable, admin rules without driver reconfiguration, automatic eviction unless tolerated.
• Device Taints and Tolerations — kubernetes.io docs: effects, rule selectors, dry-run flow, EvictionInProgress condition, feature-gate history.
• Kubernetes v1.37: Garhwal — kubernetes.io, August 2026: release announcement, DRA device taints among stable graduations.
• New Features We Find Exciting in the Kubernetes 1.37 Release — MetalBear, August 2026: DeviceTaintRule example, one-release-cycle path to stable.
• kubernetes-sigs/dra-driver-nvidia-gpu v0.4.0 release notes — NVMLDeviceHealthCheck behavior change tied to KEP-5055.
• DeviceHealthCheck: Alpha to GA — NVIDIA driver issue: planned taint schema for NVML/XID health events.
• GPU observability in Azure Kubernetes Service — Microsoft Learn: DCGM_FI_DEV_XID_ERRORS as the investigate-immediately signal, node-taint response in the pre-DRA world.