Skip to main content

Node Lifecycle Conditions Land in Kubernetes v1.37: One Status Channel for the Drain Dance

10 min readDora NodaDora Noda
Share
On this page

Ask three controllers what is happening to a Kubernetes Node and you will get three different answers reconstructed from three different clues. The drain tool watches for terminating pods. The autoscaler squints at taints. The storage operator notices maintenance only after evictions have already started. On September 9, 2026, Kubernetes v1.37 answered with a shared vocabulary: five well-known Node conditions — DrainInProgress, Drained, MaintenancePlanned, MaintenanceInProgress, and GracefulNodeShutdownInProgress — giving every component one Kubernetes-owned place to publish and read lifecycle state instead of inferring it blind.

There is a catch, and it is the whole point of this post: in v1.37 the new conditions are a status channel only. No core controller reads them yet. An administrator or an administrator-authorized controller publishes them, and the value lands the moment your own fleet automation — MachineHealthCheck remediation, drain tooling, dashboards, alerts — agrees to consume one signal instead of three guesses.

Here is what shipped, why the blind inference breaks real clusters, where lifecycle truth lives in a Cluster API fleet today, and the ownership discipline to adopt before your controllers start reading the new channel.

Five new conditions, one shared vocabulary

KEP-5683, led by SIG Node and written up by Ryan Hallisey (NVIDIA), reserves five names as well-known NodeConditionType constants. Each uses the standard condition tri-state — True while the lifecycle state is active, False when it is not, Unknown when Kubernetes cannot tell — with a stable machine-readable reason and a human-readable message:

ConditionWhat it reports
DrainInProgressThe Node is actively being drained according to the administrator's chosen drain criteria.
DrainedThe Node has reached the drain criteria selected by the administrator.
MaintenancePlannedThe Node is expected to undergo a change in the future.
MaintenanceInProgressThe Node is actively undergoing maintenance.
GracefulNodeShutdownInProgressGraceful Node Shutdown is determined to be in progress on the Node.

Note the deliberate phrasing: drain criteria are the administrator's chosen criteria. The KEP's framing — Specialized Lifecycle Management — starts from the observation that lifecycle transitions depend on context no core controller can fully define: hardware characteristics, provider integrations, application semantics. A drained node might hold zero pods or several, depending on who is asking. So Kubernetes standardizes the channel, not the definition. Your automation decides what "drained" means; the conditions give it a standard place to say so.

What v1.37 actually ships is precise and narrow. The release reserves the names and introduces the Alpha NodeLifecycleConditions feature gate, disabled by default — and in this release the gate is effectively a no-op. It does not restrict who can set the conditions, and no core component reads them. It exists so the built-in consumer behavior planned for future releases can be opted into when it arrives. You do not need to enable anything to start publishing today. An authorized maintenance controller can already write:

yaml
status:
  conditions:
  - type: MaintenancePlanned
    status: "True"
    reason: MaintenanceWindow
    lastTransitionTime: "2026-12-09T12:00:00Z"
    message: "Hardware maintenance is scheduled for this Node"

The recommended pattern is report-status-here, operate-over-there: keep using kubectl cordon, kubectl drain, taints, and workload-specific controls to change scheduling and eviction behavior, and use lifecycle conditions to make the state of that work visible to people, dashboards, alerts, and automation. Setting a condition changes nothing by itself — the reference docs are explicit that core workload controllers do not change behavior based on these conditions.

The problem: every controller reconstructs node state blind

Today, lifecycle context is scattered across readiness, taints, pod state, labels, annotations, and provider-specific APIs — and each consumer reconstructs a different picture from a different subset. One controller looks at Node readiness, another at taints, another at pods that are terminating or missing, and infrastructure providers bolt on their own labels. Those signals remain useful for their original purposes, but none of them answers the lifecycle question. A taint influences scheduling; it does not attest that a drain is in progress or that drain criteria were met. A NotReady node does not explain whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance.

When independently correct components act on incompatible reconstructions, real clusters break in ways that are miserable to debug:

  • A DaemonSet controller replaces a pod that the kubelet intentionally terminated during graceful shutdown — the controller sees an unavailable pod, not an administrator's deliberate action.
  • A Job controller waits indefinitely for a terminal pod phase on a node an administrator is already removing — nothing in the signals it reads says "this node is going away, stop waiting."
  • A storage operator learns about maintenance only after drain has already started — the evacuation it needed to orchestrate first is now racing evictions.

The Sept 9 announcement names a fourth victim explicitly: DaemonSet rollouts. A node that is broken or under maintenance still consumes the rollout's availability budget, stalling progress on healthy nodes — because the controller cannot distinguish "the new revision failed" from "an administrator took this node out of service." That distinction is exactly what MaintenanceInProgress exists to publish, and future work aims to let the DaemonSet controller consume it for rollout ordering and availability accounting.

Where lifecycle truth lives in a CAPI fleet today

For a self-hosted platform on Cluster API, this matters concretely, because a CAPI fleet already runs the drain dance constantly — every MachineDeployment rollout, scale-down, and MachineHealthCheck remediation cords, drains, and deletes nodes — and the truth about that dance lives everywhere except on the Node.

Start with remediation. A MachineHealthCheck watches a workload cluster's nodes from the management cluster and matches inferred signals: unhealthyNodeConditions entries such as Ready: Unknown or Ready: "False" held for timeoutSeconds (commonly 300), unhealthyMachineConditions on the Machine object itself, and a nodeStartupTimeoutSeconds (default 600) for nodes that never join. Short-circuiting via remediation.triggerIf.unhealthyLessThanOrEqualTo caps how much of the fleet gets remediated at once — and its default of 100% means the safety is off unless you configure it.

Every one of these inputs is a proxy: Ready: Unknown says the kubelet stopped reporting, which could be a dead host, a network partition, or a node mid-reboot for a kernel upgrade your own automation scheduled. MHC cannot tell the difference, because nothing on the Node says which one it is.

Then comes the deletion flow, which is where the richest lifecycle truth is generated — and stranded. Deleting a Machine runs roughly ten phases: pre-drain hooks, cordon plus a drain aligned with kubectl drain (evict, requeue every 20 seconds until the relevant pods are gone or nodeDrainTimeout expires), a wait for volume detach, pre-terminate hooks, deletion of the infrastructure and bootstrap objects, and finally Node deletion.

The drain itself is nuanced: DaemonSet pods, mirror pods, and anything labeled cluster.x-k8s.io/drain=skip are skipped, wait-completed pods block for completion, MachineDrainRules impose ordering, and unreachable nodes get 1-second-grace evictions. All of that state is observable — on the Machine object's DrainingSucceeded condition and in controller logs. From the workload cluster's perspective, the Node just accumulates an unschedulable taint and terminating pods, and every other consumer re-derives "draining" from those side effects.

That is the gap the new conditions close: one place on the Node to publish "drain criteria met" instead of three inferred signals scattered across two clusters' APIs. The Machine-side detail (DrainingSucceeded with its per-pod blockage messages) stays where it belongs; the Node-side summary becomes consumable by everything that watches Nodes.

The adoption playbook: who writes what

The blog post's most important operational sentence is easy to skim past: cluster administrators should decide which component owns each lifecycle condition to avoid conflicting writes. With the v1.37 gate a no-op, nothing enforces single-writer discipline — that is your job, and it is the prerequisite for trusting the channel. For a small CAPH fleet, the ownership table practically writes itself:

ConditionOwnerSets it whenClears it when
MaintenancePlannedMaintenance / upgrade automationA window is scheduledThe window starts (MaintenanceInProgress goes True) or is cancelled
MaintenanceInProgressMaintenance / upgrade automationWork starts on the nodeThe node is back in service
DrainInProgressDrain tooling (the wrapper around CAPI drain or kubectl drain)Evictions beginDrain criteria are met or the drain is aborted
DrainedDrain toolingThe fleet's drain criteria are metThe node is uncordoned and returned to service
GracefulNodeShutdownInProgressShutdown reporter (kubelet-adjacent automation)Graceful shutdown is determined in progressThe node recovers or is replaced

Four rules make the table hold:

  1. Status only, always. The conditions report; cordon, drain, taints, and pre-drain hooks act. The day a condition starts triggering behavior in your own tooling, you have built the coupling the KEP deliberately deferred — document it as yours, not Kubernetes'.
  2. If you set it, you clear it. A stale Drained: True on a node back in service is worse than no signal, because consumers will trust the channel once it exists. Pair every set path with a clear path, including the abort path (cancelled window, failed drain).
  3. Stable reasons, human messages. Keep reason values from a small fleet-wide vocabulary (MaintenanceWindow, CapiRemediation, KernelUpgrade) so alerts and dashboards can match on them; put the specifics — ticket, window, criteria — in message.
  4. Dashboards and alerts consume first, controllers second. The immediate payoff needs no control loop: MaintenancePlanned in your fleet board, DrainInProgress duration alerts, "nodes with Drained: True older than N hours" hygiene checks. Promote a signal to controller input only after watching it stay correct for a few maintenance cycles.

Note what this unlocks for MHC specifically: today a remediation loop cannot distinguish "node unexpectedly unhealthy" from "node unhealthy because we are working on it." Publishing MaintenanceInProgress before maintenance starts gives your on-call — and eventually your own controllers — the context to read a Ready: Unknown correctly instead of racing the maintenance with a remediation.

What v1.37 doesn't do (yet)

Honesty first, because the scope is the story:

  • Nothing core consumes the conditions. No scheduler, workload controller, or kubelet behavior changes when you set them. The DaemonSet rollout accounting sketched in the announcement is future work, explicitly requiring careful design first.
  • There is no ownership or locking API. Single-writer discipline is convention plus RBAC today. Longer-term coordination may need explicit ownership, locking, or a dedicated API — the Node Lifecycle Working Group names that as open design, alongside a NodeMaintenance-style request API the community has been sketching in parallel.
  • Drained attests your criteria, not emptiness. A drained node may legitimately still run DaemonSet pods, mirror pods, or skipped workloads. Consumers must treat it as "the publisher's bar was met," never as "zero pods."
  • The alpha gate gates nothing yet. Enabling NodeLifecycleConditions changes no behavior in v1.37; it is a reservation for future consumer behavior you will opt into later.

None of that diminishes the release — it defines how to use it. A shared vocabulary is valuable the moment two of your systems agree on it, and the cost of publishing five conditions from automation you already run is close to zero. The expensive failure is the one the KEP describes: every component inventing its own inference, then acting on incompatible conclusions at 3 a.m. during a kernel rollout.

Follow the follow-ups through KEP-5683 and the Node Lifecycle Working Group (with SIG Node and SIG Apps), where the consumer behaviors — DaemonSet availability accounting, graceful-shutdown coordination, drain awareness — are being designed next. The fleet that publishes lifecycle state today is the fleet whose controllers can consume it tomorrow.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex