Node Readiness Controller Goes Alpha: Declarative Taint Gating for GPU and Driver Readiness

Every GPU node in your fleet lies to the scheduler on every boot. Kubelet flips the node to Ready the moment it can talk to the API server — while the NVIDIA driver container is still compiling kernel modules, the CNI agent is still programming routes, and the storage driver is still attaching volumes. Pods land in that window and pay for it: CrashLoops, silent network failures, evictions. NVIDIA's own Network Operator docs admit the failure mode outright — during driver load, "workloads might get scheduled on that node" and pods using NVIDIA NICs "might silently fail or hang" when the driver reloads underneath them.

The fix the ecosystem has wanted for years finally has a standard shape: the Node Readiness Controller, announced on the Kubernetes blog in February 2026 and now a kubernetes-sigs project, lets operators define per-node-group scheduling gates so a node only drops its not-ready taint once its infrastructure dependencies are actually verified.

Here is the verdict up front: the mechanism is sound, the dry-run-then-enforce rollout ladder makes it safe to trial, and a self-hosted fleet should adopt it in bootstrap-only mode on GPU pools now — while waiting for the API to graduate before betting continuous enforcement on it. The rest of this post shows the gate YAML, how it compares to the half-measures you already run, and exactly where the alpha rough edges are.

"Ready" means the kubelet is up, not that the node can run your pod​

The gap is architectural, not a bug anyone forgot to fix. The core Ready condition answers one question — "is the kubelet healthy and reporting?" — and the scheduler treats a True as permission to bind. Everything else a pod actually needs is somebody else's problem:

• CNI agents (Cilium, Calico) need seconds to minutes after kubelet start to program networking. Karpenter learned this the hard way: pods bound before Cilium finished setup failed because pod IP assignment wasn't ready, which is why Karpenter grew startupTaints in the first place.
• GPU drivers install as containers via the GPU Operator — driver compile, toolkit setup, device-plugin registration, validator pass — a multi-stage pipeline during which nvidia.com/gpu capacity flickers and nvidia-smi inside a pod can fail with "couldn't communicate with the NVIDIA driver."
• Storage and network drivers attach late and reload dangerously. The NVIDIA Network Operator's OFED upgrade path explicitly tells operators to cordon and drain before the driver reloads, because running pods lose their interfaces mid-flight.

Each vendor solved this privately with its own taint — the EBS CSI driver, for example, clears an ebs.csi.aws.com/agent-not-ready:NoExecute taint when it becomes healthy, and Karpenter treats taint removal as the signal that a NodeClaim is initialized. What never existed was the generic version: a declarative way to say "this class of nodes is schedulable only when conditions X, Y, and Z all hold," enforced by one controller instead of N vendor-specific hacks.

What the controller actually does​

The Node Readiness Controller, first built by Ajay Sundar Karuppasamy at Google as a standalone gate controller and since migrated into kubernetes-sigs, centers on one CRD: NodeReadinessRule (readiness.node.x-k8s.io/v1alpha1). A rule names a set of node conditions that must hold, a taint to assert while they don't, a label selector for the node group it governs, and an enforcement mode. The controller watches node conditions and adds or removes the taint — nothing more.

The key design decision is what the controller doesn't do: it never probes anything itself. It reacts to NodeCondition signals published by whatever already observes the node — Node Problem Detector with a custom plugin, a vendor daemon, or the project's own lightweight Reporter agent that polls a local HTTP endpoint and patches the condition. That decoupling is what makes it a platform primitive rather than another monitoring tool: the health signal and the scheduling enforcement evolve independently, and a fleet can adopt gating without replacing its existing detectors.

The gate, concretely: a GPU pool that waits for its driver​

Enough mechanism — here is the artifact. A GPU node pool where inference pods must never land before the driver stack is verified gets a rule like this:

apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: gpu-driver-verified
spec:
  conditions:
    - type: "gpu.example.com/DriverVerified"
      requiredStatus: "True"
    - type: "gpu.example.com/DevicePluginReady"
      requiredStatus: "True"
  taint:
    key: "readiness.k8s.io/gpu-not-ready"
    effect: "NoSchedule"
    value: "pending"
  enforcementMode: "bootstrap-only"
  nodeSelector:
    matchLabels:
      node-pool: gpu-inference

All conditions in a rule must hold (AND semantics) before the taint lifts, and the nodeSelector scopes the gate to the GPU pool — CPU pools follow their own rules, or none at all. The two conditions come from wherever you already observe driver health: an NPD custom plugin that shells out to nvidia-smi, the Reporter agent polling the device plugin's health endpoint, or a DaemonSet that patches the condition after the validator passes. The official docs' canonical example gates a worker pool on a CNIReady condition with the same shape, so a networking gate and a driver gate compose as two rules with two taints rather than one tangled script.

Note the deliberate choice of bootstrap-only here — that needs a section of its own, because mode selection is where teams will get this wrong.

Bootstrap-only vs continuous vs dry-run: pick the mode before you pick the fight​

The controller ships three postures, and they answer three different questions:

The safe adoption ladder runs top to bottom. Dry-run first, always: a new readiness rule is a fleet-wide scheduling change wearing a YAML costume, and dry-run shows the affected nodes in the rule's status without moving a single pod. Promote to bootstrap-only for init-time gates — this is the mode with the smallest blast radius, since a misconfigured gate delays scheduling on fresh nodes rather than evicting running work. Reserve continuous for dependencies whose mid-life failure genuinely demands re-tainting, and remember its effect is NoSchedule, not NoExecute: it stops new placements; it does not evict what already runs. If you need eviction on driver failure, that is still a problem-detector-plus-drain automation, not this controller.

What you already have, and where each stops short​

Honest accounting matters here, because most fleets already run two or three partial answers:

Two rows deserve emphasis. NPD plus this controller is the intended pairing, not a rivalry: NPD observes, the controller enforces. And v1.37's new lifecycle conditions (DrainInProgress, MaintenancePlanned, and friends) are complementary vocabulary arriving from SIG Node — standard condition names the controller's rules can eventually key on, once core components actually publish them. Neither replaces per-node-group condition gating.

Is an alpha sigs project safe to gate scheduling on?​

Now the question the title promised. The candid answer has three parts: what "alpha" concretely means here, what the known rough edges are, and what staged adoption looks like.

What alpha means: the API group is v1alpha1, the project cut its v0.1.0/v0.1.1 releases around the February announcement, and the maintainers are explicit that they want community feedback to shape the roadmap (they ran the idea through a KubeCon NA 2025 unconference and a KubeCon EU 2026 maintainer session titled "Addressing Non-Deterministic Scheduling" first). Expect API churn before any beta: write rules you can regenerate, not hand-tuned snowflakes.

Known rough edges, from the project's own tracker: enable the validation webhook — without it, conflicting rules can fight over the same taint key and flap a node in and out of schedulability. Bootstrap-completion bookkeeping has had edge cases around rule deletion and recreation leaving stale state that lets nodes skip evaluation. And continuous mode is young enough that operators are still filing the obvious feature requests (rule suspension, heartbeat freshness for agent-reported conditions). None of these is disqualifying; all of them argue for starting in dry-run and bootstrap-only, where a controller bug delays a fresh node instead of flapping a live one.

The staged verdict for a small self-hosted fleet:

1. Now: install the controller with the webhook enabled, write one dry-run rule for your GPU pool's driver-verified condition, and watch the rule status for a full node-rotation cycle. Cost: near zero. Risk: zero — dry-run moves nothing.
2. Next: promote the GPU gate to bootstrap-only. This closes the real failure mode — inference pods landing on driverless nodes — with the smallest possible blast radius.
3. Later: graduate to continuous for driver/agent health only after the API moves past v1alpha1 and your dry-run history shows the conditions are trustworthy. A flaky condition in continuous mode is a self-inflicted scheduling outage, so the signal must earn enforcement.

That is a yes with a sequencing constraint, not a "wait for GA." The failure mode it closes is happening to your GPU nodes today; the dry-run ladder means you can prove the fix before it touches scheduling.

What this means for a Cluster API fleet on owned machines​

For a CAPI-managed fleet — Hetzner machines behind the Hetzner provider, say — the payoff is structural. CAPI already gives you declarative machine lifecycle: MachineDeployments own pools of machines with labels you control. The readiness controller slots underneath that layer without touching the provisioner at all: each pool's labels become a rule's nodeSelector, so the GPU MachineDeployment gets a driver-verified gate, the general worker pool gets a CNI gate, and a storage-heavy pool gets a CSI gate — each pool following its own readiness path while CAPI keeps doing machine lifecycle above.

This is also the shape of fleet provisioning discipline maturing: declare the desired state of every machine, including what "ready to take work" means for its class, and let controllers reconcile toward it. CAPI reconciles the machines; the readiness controller reconciles their schedulability. No bootstrap scripts that sleep 120 and hope the driver finished, no per-pool snowflake DaemonSets whose only job is to remove a taint they invented.

The scheduler is about to get pickier, and that's good​

Zoom out and the direction is unmistakable. SIG Node is standardizing the vocabulary (KEP-5683's lifecycle conditions), the device ecosystem is moving scheduling intelligence into DRA's claim-based model, and now a sigs-owned controller turns node conditions into scheduling enforcement with dry-run safety built in. The era of "kubelet says Ready, good luck" is ending — replaced by per-pool, declarative definitions of what ready actually means.

Start with dry-run on your GPU pool this week. Your inference pods will never know the gate is there, which is exactly the point: the best scheduling fix is the one that turns a 3 a.m. CrashLoop page into a node that quietly waited for its driver instead of scheduling into the gap.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.