Skip to main content

Kubernetes v1.37's Node Lifecycle Conditions: Teaching MachineHealthCheck to Tell 'Draining' Apart From 'Dying'

9 min readDora NodaDora Noda
Share
On this page

Picture the scene: your on-call gets paged at 03:00 because a tenant workload lost a node. The timeline shows the machine was cordoned, drained, and halfway through a planned kernel upgrade when your remediation controller decided the node looked dead and deleted the Machine out from under the maintenance job. The node wasn't dying. It was draining. And nothing in the Kubernetes API could tell your automation the difference.

Kubernetes v1.37, released August 26, 2026 as "Garhwal," starts fixing exactly that. A September 9 follow-up post from SIG Node introduces five well-known Node Lifecycle Conditions — a shared, Kubernetes-owned vocabulary for saying "this node is draining," "this node is under maintenance," or "this node is shutting down gracefully." Here is the honest verdict up front: in v1.37 these conditions are report-only. The alpha gate is off by default and no core component reads them yet. But the vocabulary is the part your Cluster API fleet should adopt now, because the single most valuable signal in the set — MaintenancePlanned — is the "hold remediation, this is planned work" flag that MachineHealthCheck has never had.

ConditionMeaningWhat it tells your MHC flow
DrainInProgressNode is actively being drained per the admin's criteriaAttests what drain tooling already knows; useful for dashboards, not new to remediation
DrainedNode has met the admin's drain criteriaSame — an attestation, not a trigger
MaintenancePlannedNode is expected to change in the futureGenuinely new: the hold-remediation signal MHC lacks today
MaintenanceInProgressMaintenance work is underwayDistinguishes planned work from failure during the window
GracefulNodeShutdownInProgressGraceful Node Shutdown is in progressExplains a NotReady that is orderly, not a crash

The rest of this post maps each of those rows against what MachineHealthCheck already does, shows what a Cluster-API-managed fleet should start publishing this quarter, and names what still has to arrive before remediation can key on the new signals.

The taint-and-annotation soup nodes live in today

Kubernetes already has many ways to describe what is happening on a node: Readiness, taints, pod state, labels, annotations, and provider-specific APIs. The problem, in SIG Node's own framing, is that each one exposes only part of the picture, and every consumer reconstructs lifecycle state from a different mix of indirect signals.

Concretely, a fleet operator triaging an unhealthy node today pieces together clues like these. A NotReady condition with a fifteen-minute duration is the classic MachineHealthCheck trigger — but it cannot say whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance. The node.kubernetes.io/unschedulable taint says no new pods should land here, yet as the upstream post notes, a taint influences scheduling without attesting that a drain is in progress or that drain criteria were met. Node-problem-detector adds custom conditions for kernel-level faults, which is genuinely useful failure data but says nothing about administrative intent. And every infrastructure provider layers on its own labels or annotations, which are opaque to every controller except the one that wrote them.

The failure mode this produces is specific and expensive: remediation that fights maintenance. A MachineHealthCheck watching Ready: Unknown for longer than nodeStartupTimeout fires its remediation template — typically deleting the Machine so Cluster API provisions a replacement — while a human or an upgrade controller is mid-drain on that exact node. Two controllers, both correct by their own lights, working against each other. One controller might look at readiness, another at taints, another at terminating pods, and none of them can answer the one question that matters: is somebody already handling this node on purpose?

What v1.37 actually ships: vocabulary first, consumers later

The September 9 post, part of the v1.37 "Garhwal" release series and led by SIG Node with the Node Lifecycle Working Group and SIG Apps, reserves five names as well-known NodeConditionType constants under KEP-5683. The semantics are deliberately narrow. DrainInProgress means the node is actively being drained according to the administrator's chosen drain criteria; Drained means those criteria have been met. MaintenancePlanned carries a message like "hardware maintenance is scheduled for this node" with a timestamp; MaintenanceInProgress marks the window itself. GracefulNodeShutdownInProgress reports that graceful shutdown is underway.

Three facts about the rollout matter more than the names. First, v1.37 introduces the alpha NodeLifecycleConditions feature gate, disabled by default — and in this release the gate is effectively a no-op. It does not restrict who may set the conditions, and no core component reads them. You do not need to enable anything to start publishing them today.

Second, the recommended pattern is report status, don't operate through them. Keep changing scheduling and eviction behavior with the existing mechanisms — kubectl cordon, kubectl drain, taints, workload-specific controls — and use the conditions to make the state of that work visible to people, dashboards, alerts, and automation that opts into the signal. A taint remains the thing that moves pods; the condition is the thing that explains why.

Third, the canonical publisher flows are split exactly along the drain-versus-maintenance line. Maintenance automation sets MaintenancePlanned when a window is scheduled, then flips to MaintenanceInProgress when work starts. Drain automation sets DrainInProgress when it begins evicting pods and Drained when the administrator's criteria are satisfied. That split is the whole ballgame for remediation, because only one half of it is information your fleet doesn't already have somewhere.

The overlap map: what MHC already covers versus what's genuinely new

Cluster API's MachineHealthCheck decides "replace this node" from its own condition watch. The current API — unhealthyNodeConditions in cluster.x-k8s.io/v1beta2, renamed from unhealthyConditions during the v1.11 cycle — matches node conditions against duration thresholds, backed by nodeStartupTimeout (commonly ten minutes, the grace period for a fresh node to register before it counts as unhealthy) and maxUnhealthy (commonly around forty percent, the circuit breaker that stops remediation from deleting half the fleet at once). The textbook rule is Ready: Unknown sustained past the timeout: remediate.

Here is each new condition measured against that machinery:

  • MaintenancePlanned — genuinely new, and the most valuable row. Nothing in MHC's model expresses "don't remediate this node next Tuesday; we already know." nodeStartupTimeout covers slow boots, not scheduled future work. A maintenance controller publishing MaintenancePlanned with a window timestamp gives every consumer — MHC, alerting, dashboards — a forward-looking hold signal that currently has no equivalent anywhere in the node API.
  • MaintenanceInProgress — new during the window. Today a node mid-maintenance looks identical to a node mid-failure: NotReady, possibly unschedulable, pods evicting. The condition separates "planned work is happening" from "something broke," which is precisely the distinction a remediation controller needs to stand down instead of deleting the Machine.
  • GracefulNodeShutdownInProgress — explanatory, partially overlapping. A kubelet doing an orderly shutdown already behaves better than a crashed one, but to MHC both still read as NotReady aging toward the timeout. The condition doesn't change the remediation decision by itself; it changes the confidence. A short shutdown window that resolves on its own should never have counted down the same fuse as a dead box.
  • DrainInProgress and Drained — attestations, not triggers. This is the honest caveat: drain automation already knows it is draining. These conditions don't tell the drain tool anything new. Their value is outward — a standard place for other controllers to read drain state instead of scraping tool-specific annotations. For MHC specifically, a node carrying Drained alongside an Unschedulable taint is a node somebody is handling; remediation should treat it as claimed work, not a corpse.

The medik8s ecosystem is the parallel to watch. Its Node Healthcheck operator already watches NodeConditions and instantiates remediation custom resources — reboot via Self Node Remediation, fencing, or machine deletion — from condition matches. That is the exact consumer shape KEP-5683's authors are inviting: condition publishers on one side, remediation controllers matching on them on the other. When MHC or NHC-style controllers learn to match MaintenancePlanned as a hold and Drained as claimed work, the 03:00 page from the introduction stops happening.

What a Cluster-API fleet should do this quarter

The good news is that the useful work requires no upgrade and no feature gate. Publishing conditions is plain API writes, available today.

First, make your maintenance automation a condition publisher. Whatever drives your kernel upgrades, MachineDeployment rolls, or hardware windows — write MaintenancePlanned with a reason and message naming the window when the work is scheduled, and flip to MaintenanceInProgress when it starts. Clear both when the node is healthy again. This costs a few lines in the job that already exists and immediately improves every dashboard and alert that reads node status.

Second, keep remediation keyed on the existing signals. Do not add the new conditions to unhealthyNodeConditions yet — no core component honors them, and matching remediation on a signal nothing authoritative sets is how you build a trigger that fires on stale data. Keep the NotReady-duration rule, the startup timeout, and the maxUnhealthy breaker exactly where they are.

Third, teach your alerting the new vocabulary before your controllers. A page that says "node NotReady 16m, MaintenanceInProgress since 02:58" is a page the on-call can acknowledge and go back to sleep for. That alone justifies publishing.

Fourth, track KEP-5683 for the consumer side. The upstream post explicitly invites maintainers to bring use cases to the Node Lifecycle Working Group, SIG Node, and SIG Apps. The conditions graduate from vocabulary to mechanism the moment controllers — MHC, medik8s NHC, autoscalers — agree on what each one implies. A small fleet that already publishes will inherit that behavior; a fleet that waited will be retrofitting publishers under time pressure.

Vocabulary now, mechanism later

The arc here is familiar from Kubernetes history: first the API reserves the words, then the ecosystem learns to speak them, then core controllers start listening. Taints went through the same journey from advisory marks to eviction machinery. Node Lifecycle Conditions are at stage one — five reserved names, an alpha gate that is deliberately a no-op, and a clear instruction to report rather than operate.

For a self-hosted PaaS running tenant workloads on machines it owns, the payoff is concrete even at stage one. The difference between "draining" and "dying" is currently reconstructed from clues by every controller independently, and every reconstruction can be wrong in the expensive direction — deleting a machine that was being cared for. A shared condition both sides can read turns that guess into a lookup. Publish the maintenance half now, keep remediation where it is, and be ready when the consumers arrive.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.

Related articles

Run this on infrastructure you own

bex is the open-source, AI-native Render alternative — push a git repo and get a running HTTPS service on your own machines.

Get started with bex