Skip to main content
bex.co
419
Blog

Kubernetes' New Node Readiness Controller Closes the 'Pod Scheduled Before the Node Actually Works' Gap

8 min readDora NodaDora Noda
Self-Hosting
PaaS
Infrastructure
Engineering
Share
On this page

A HetznerBareMetalHost joins a Cluster API cluster, kubelet checks in, the API server marks the node Ready, and the scheduler does exactly what it's supposed to: it places pods on it. The trouble is "Ready" means kubelet is running and the API is reachable — nothing more. The CNI DaemonSet hasn't finished rolling out. The CSI driver hasn't registered a CSINode object. Pods land anyway, fail to get an IP or a volume, and crash-loop while a burst of image pulls hammers the root disk's IOPS, which slows down the very DaemonSets the node is waiting on.

That gap between "kubelet is up" and "the node can actually do the things a pod needs" has a name now. In February 2026, kubernetes-sigs/node-readiness-controller shipped as an out-of-band implementation of KEP-5233 (NodeReadinessGates): a NodeReadinessRule CRD that taints a node until specific conditions go True, then untaints it — no scheduler changes, no kubelet patch, deployable on any cluster today. The short version of what it does: instead of one binary Ready flag, you get as many readiness gates as you have dependencies, each backed by a declarative rule instead of a bespoke controller. Below is exactly what wiring three of those gates — cloud-controller-manager, CNI, CSI — into a Hetzner bare-metal bring-up sequence looks like, and why a managed EKS or GKE node pool mostly never needs to have this conversation.

Why Managed Node Pools Never Feel This​

On EKS, the VPC CNI plugin ships baked into the AMI and starts before kubelet ever registers the node. On GKE, the CNI and CSI equivalents are managed add-ons with their own SLA, provisioned as part of the node pool's bootstrap script — by the time kubelet reports Ready, networking is already live in the overwhelming majority of cases. Cluster autoscaler on both platforms also models node-bring-up latency into its scale-up math, so a freshly added node doesn't immediately become a scheduling target for pending pods; there's a buffer baked into the scale-up path itself.

None of that exists on a bare-metal fleet provisioned by Cluster API Provider Hetzner (CAPH). A HetznerBareMetalHost goes through PXE-style provisioning via the Hetzner Robot API, cloud-init runs, kubeadm joins the node to the cluster — and only after that does an operator apply the Hetzner Cloud Controller Manager, a CNI (Cilium or Flannel), and the Hetzner CSI driver as separate manifests.

CAPH's own docs acknowledge the resulting window explicitly: "it is normal for workload nodes to remain in NotReady state" until CNI and CCM are installed, and the documented fix for the CCM stage is a manual kubectl patch adding a toleration for node.cloudprovider.kubernetes.io/uninitialized to the CNI DaemonSet so it can even start. That's a taint an operator has to know about and patch around by hand, stage by stage, with no equivalent for CSI at all. On a fleet with a handful of bare-metal nodes rather than an autoscaled pool numbering in the hundreds, every one of those minutes-wide gaps is a bigger fraction of total capacity — and it's fully exposed to the scheduler instead of hidden behind a managed bootstrap sequence.

The Workaround This Replaces​

The standard fix predates NRR by years: register the kubelet with --register-with-taints, then run a custom controller that watches for readiness signals and strips the taint once satisfied. It works, but every cluster ends up with its own bespoke version of that controller, granted broad RBAC (it needs to patch node objects and taints), and prone to a race between the taint-removal write and the scheduler's read of node state — the same shape of bug that shows up in issues like kubernetes/kubernetes#72129. It's a real fix, but it's homegrown infrastructure that every self-hosted operator ends up rebuilding independently.

What NodeReadinessRule Actually Looks Like​

The controller centers on one CRD. The canonical shape from the project's announcement:

yaml
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: network-readiness-rule
spec:
  conditions:
    - type: "cniplugin.example.net/NetworkReady"
      requiredStatus: "True"
  taint:
    key: "readiness.k8s.io/network-unavailable"
    effect: "NoSchedule"
  enforcementMode: "bootstrap-only"
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/worker: ""

The controller watches for the listed condition types on matching nodes, applies the taint the moment a node registers, and removes it only once every listed condition reports True. Two fields matter most for a bring-up sequence: enforcementMode: bootstrap-only fires once and stops watching (the right choice for one-time initialization, as opposed to continuous, which keeps re-tainting a node if a dependency later degrades), and a validation webhook rejects rules that would leave conflicting taints on the same node — the exact class of bug that plagued the hand-rolled version.

Wiring It Into a CAPH Bring-Up Sequence​

Here's the gap the announcement doesn't cover: none of CCM, CNI, or the Hetzner CSI driver natively write a Node Condition. NodeReadinessRule gates on conditions that already exist on the object — it doesn't check DaemonSet rollout status or CSINode registration itself. To use it on a CAPH fleet, each stage needs a small reporter patching the corresponding condition, which the NRR project ships as an ecosystem piece for exactly this purpose: a lightweight readiness-condition-reporter that runs an HTTP or process check and patches a Node Condition from the result. That reporter is the piece an operator has to deploy and configure — it isn't automatic, and treating it as a given would be dishonest about what "wiring this in" costs.

With reporters in place, three rules map onto CAPH's actual three-stage bootstrap, ordered to match the real dependency chain:

yaml
# Stage 1 — gate on the Hetzner Cloud Controller Manager finishing node initialization
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: caph-ccm-readiness
spec:
  conditions:
    - type: "hetzner.syself.com/CCMInitialized"
      requiredStatus: "True"
  taint:
    key: "readiness.k8s.io/ccm-not-ready"
    effect: "NoSchedule"
  enforcementMode: "bootstrap-only"
yaml
# Stage 2 — gate on the CNI DaemonSet (Cilium or Flannel) reporting ready
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: caph-cni-readiness
spec:
  conditions:
    - type: "hetzner.syself.com/CNIReady"
      requiredStatus: "True"
  taint:
    key: "readiness.k8s.io/cni-not-ready"
    effect: "NoSchedule"
  enforcementMode: "bootstrap-only"
yaml
# Stage 3 — gate on the Hetzner CSI driver's CSINode registration
apiVersion: readiness.node.x-k8s.io/v1alpha1
kind: NodeReadinessRule
metadata:
  name: caph-csi-readiness
spec:
  conditions:
    - type: "hetzner.syself.com/CSIRegistered"
      requiredStatus: "True"
  taint:
    key: "readiness.k8s.io/csi-not-ready"
    effect: "NoSchedule"
  enforcementMode: "bootstrap-only"

Each reporter check maps directly to something CAPH's docs already describe as a manual verification step: the CCM reporter polls for the node's node.cloudprovider.kubernetes.io/uninitialized taint clearing (the same signal an operator currently checks by hand), the CNI reporter checks the Cilium or Flannel DaemonSet's status.numberReady against desiredNumberScheduled on that node, and the CSI reporter polls the API server for a CSINode object listing the Hetzner driver.

All three rules run bootstrap-only, because none of these are expected to flap after initial bring-up — that's exactly the case the mode exists for. The result: a node accepts zero pods until CCM, CNI, and CSI have all confirmed ready, in that order, with three declarative rules and a validation webhook standing between the fleet and the conflicting-taint bugs the old hand-rolled controller was prone to.

Testing Before You Trust It​

Every rule above also supports a dryRun: true field, which is the sane way to introduce this on a fleet that's already serving traffic. In dry-run, the controller evaluates conditions and logs exactly which nodes would be tainted or untainted, and writes that verdict into the rule's status — without touching a single taint. That matters specifically on bare metal, where a wrongly-scoped rule (say, a nodeSelector that also catches control-plane nodes, or a condition type the reporter never actually sets) can otherwise strand every future node addition in a permanently-tainted, permanently-unschedulable state with no pods landing and no obvious error. Running each of the three CAPH rules in dry-run against a couple of freshly-provisioned HetznerBareMetalHost nodes first — confirming the CCM, CNI, and CSI reporters flip their conditions in the right order, in the expected few minutes — is what actually derisks the second problem this piece opened with: a badly-configured readiness gate is just as capable of leaving a node stuck as no gate was of leaving it half-ready.

Rollout Path for a Small Fleet​

To be clear about what this does and doesn't solve: NRR removes the bespoke taint-removal controller and its broad RBAC footprint, and it removes the race between taint removal and scheduler reads. It does not remove the need to deploy something that reports conditions in the first place — the readiness-condition-reporter (or an equivalent Node Problem Detector plugin) is still infrastructure an operator owns and maintains per fleet. And at v0.3.0, alpha status, this belongs on a non-production bare-metal fleet first; the project's own roadmap lists metrics integration and large-scale performance work as still outstanding.

A reasonable order for a CAPH operator to actually adopt this: install the controller and its validation webhook, deploy the three reporters against a staging HetznerBareMetalHost pool, apply all three rules with dryRun: true and watch a handful of real node joins go through the CCM → CNI → CSI sequence in the rule status, then flip dryRun off for the CCM rule alone (the lowest-risk of the three, since it gates against a taint CAPH already documents) before extending to CNI and CSI. That's a slower rollout than reaching for the old hand-rolled controller, but the payoff is that once it's live, adding or replacing a bare-metal node stops being an event that needs a human watching a kubectl get nodes -w window to confirm nothing landed on it too early.

That's a fair trade for a fleet where "half-ready node accepts a pod" isn't a once-a-quarter annoyance absorbed by autoscaler slack, but a bug that shows up every time a bare-metal host gets added or replaced.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own, provisioned through Cluster API. Fleets like this are exactly where a declarative node-readiness gate earns its keep. Star the repo on GitHub or deploy your first app today.

Sources​

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex
Back to all posts