Eleven Minutes, Zero Humans: What Kairos's Self-Healing Upgrades Teach a Bare-Metal PaaS

Once upon a time, upgrading a Kubernetes control plane meant staying awake for it. SSH into every node. Run the upgrade by hand. Watch etcd health the whole time, hoping quorum holds through every reboot.

On July 7, 2026, that story ended — at least for one homelab that decided to stop trusting humans with reboots. Eleven minutes wall-clock. Zero humans running kubectl apply after the merge. Three control-plane nodes rolling through an OS and Kubernetes upgrade without breaking quorum or disturbing a single workload. The receipts are public, the bugs that almost wrecked it are documented, and the pattern translates directly to the problem every bare-metal PaaS still solves by hand: how do you roll hundreds of nodes when no one is watching the screen.

Verdict up front. Kairos's A/B atomic upgrades plus a six-tool GitOps pipeline delivered a genuinely hands-off upgrade. The two-line diff that triggered it was trivial. The two bugs that almost made it catastrophic were not — and they are exactly the guards a Hetzner + Cluster API fleet needs to steal before it claims "self-healing" on its own status page.

What the CNCF Pipeline Actually Is​

The post is CNCF's August 14, 2026 Kubestronaut entry by Olivier Calzi — a Golden Kubestronaut building the "root" of a platform that will keep growing: more tooling, more workloads, more clusters hanging off it. He bootstrapped the management cluster with OpenTofu, three control-plane nodes in high availability, k3s as the distribution, Cilium as the CNI — all provisioned as code before a workload ever ran. The declared goal was not more features, but a simple, reliable upgrade process that survives the 2026 pace of CVEs without relearning the runbook every time.

Three of those nodes run Kairos Hadron, an immutable Linux distribution built around two ideas that change the upgrade math: every OS image is a signed OCI container, and every upgrade writes to an inactive partition and reboots into it. There is no apt upgrade mutating a running root. Rollback is not a restore procedure — it is booting the other partition again.

Six tools, each with one job, form the pipeline that made the July 7th upgrade to Hadron v0.4.0 a non-event:

The two-line diff that triggered all of it:

--- a/upgrades/mgmt/hadron-upgrade.yaml
+++ b/upgrades/mgmt/hadron-upgrade.yaml
 metadata:
-  name: hadron-mgmt-v0-3-0
+  name: hadron-mgmt-v0-4-0
 spec:
-  image: quay.io/kairos/hadron:v0.3.0-standard-amd64-generic-v4.1.1-k3s-v1.35.5-k3s1
+  image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v4.1.2-k3s-v1.35.5-k3s1

Why the name has to change: NodeOpUpgrade is a one-shot resource. Once the operator marks it complete it never reprocesses the same object. Patching spec.image on an existing CR does nothing — the only way to trigger a new rollout is a new object.

Real timestamps from that run, as published:

• Total wall-clock: 11 minutes for three control-plane nodes, sequential
• Human intervention after merge: 0 — ArgoCD applied, operator executed
• etcd quorum broken: never
• Workload disruption: none reported
• Concurrency: 1 node at a time (after the fix described below)

That last line is the whole lesson. The pipeline worked in 11 minutes because one integer was not zero.

The Two Bugs That Almost Broke It​

The post is unusually honest about where automation quietly did the wrong thing while looking correct. Both bugs would have been invisible in a walkthrough that only shows the happy path.

Bug 1: concurrency: 0 means "all at once"​

Kairos's NodeOpUpgrade spec has a concurrency field controlling how many nodes reboot simultaneously. Calzi assumed 0 meant "one at a time." It means the opposite — all nodes at once. During a homelab test, three control-plane nodes rebooted simultaneously. With three members, losing all three at the same instant should have taken etcd quorum down. It survived by luck, not design.

The fix is one integer:

# before — dangerous default
spec:
  concurrency: 0   # actually: all nodes simultaneously

# after — the backbone of the 11-minute run
spec:
  concurrency: 1   # one node reboots, rejoins, then the next

For any PaaS fleet, this is the first guard to steal: a rollout that cannot guarantee sequential control-plane reboots is not hands-off, it is hands-off-quorum. On a Hetzner fleet managed by Cluster API's KubeadmControlPlane, the equivalent knob is rolloutStrategy.rollingUpdate.maxSurge and the etcd quorum guard that KCP enforces — but the principle is identical. If your spec can express "all at once," someone will ship it.

Bug 2: Renovate bumped the image but not the CR name — and nothing happened​

Renovate's custom regex manager was configured to watch quay.io/kairos/hadron tags and bump the upgrade CR. The config originally used a field called extractVersionTemplate to translate the dash-formatted version in metadata.name (v0-3-0) into a comparable SemVer (v0.3.0). That field does not exist in Renovate's custom manager schema. Renovate silently ignored it.

Result: Renovate correctly opened a PR bumping spec.image to the new tag, but left metadata.name unchanged at hadron-mgmt-v0-3-0. The operator saw no new CR — because there was no new object — and did nothing. The pipeline looked broken. It was actually half-executed, which is worse.

Fix: switch to currentValueTemplate, which Renovate's custom managers actually evaluate, so the dash-to-dot conversion works and both lines bump together. The author keeps the human review step in the pipeline for exactly this reason: not to execute the upgrade, but to catch the one place where automation quietly did half the job while reporting success.

Both bugs share a shape a fleet operator should recognize: a field that silently defaults to the dangerous value, and a field that silently does nothing when misspelled. Neither failed loudly. Both needed a human reading the diff to notice.

Why A/B Makes Self-Healing Possible​

The reason the pipeline can afford to be hands-off is not just GitOps. It is the disk layout.

A traditional mutable node upgrades in place: apt update overwrites the running root, and a failed upgrade leaves the root in an unknown, half-written state that requires human triage. Rollback is a second, distinct procedure with its own failure modes.

Kairos (and Flatcar, Talos, and Fedora CoreOS in different variations) inverts this:

Calzi's pipeline adds one more future guard: a CI dry-run stage running kairos-agent upgrade --recovery against the new image before the PR ever merges, catching a bad image before it touches a live node. The recovery partition is kept distinct from the active slot by design — the operator never upgrades both at once.

The self-healing claim is therefore not a Kubernetes reconciliation loop. It is a bootloader policy: a node that cannot boot the new slot boots the old one, rejoins the cluster, and becomes eligible again. The Kairos operator's job is to sequence that behavior one node at a time, not to invent it.

Translating the Pattern to a Hetzner + Cluster API Fleet​

A Kairos homelab with three nodes and a PaaS fleet with hundreds of Hetzner machines do not share a scale, but they can share a control loop. Here is the mapping that turns "11 minutes on three nodes" into a fleet rollout an operator can trust at 3 a.m.

Concrete shapes, side by side:

# Kairos-native: one node reboots into a new A/B slot
apiVersion: operator.kairos.io/v1alpha1
kind: NodeOpUpgrade
metadata:
  name: hadron-mgmt-v0-4-0
spec:
  image: quay.io/kairos/hadron:v0.4.0-standard-amd64-generic-v4.1.2-k3s-v1.35.5-k3s1
  concurrency: 1
  nodeSelector:
    matchLabels:
      kairos.io/managed: "true"

# CAPI-native: one MachineDeployment rolls to a new infrastructure template
apiVersion: cluster.x-k8s.io/v1beta2
kind: MachineDeployment
metadata:
  name: fleet-workers
spec:
  replicas: 20
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      infrastructureRef:
        name: hetzner-workers-v4  # bumped by Renovate, gated by policy
      bootstrap:
        configRef:
          name: workers-bootstrap-v4

The gateway cluster in Calzi's post — a single Kairos node running Netbird — now rides the same GitOps loop as the management cluster. That matters for fleet thinking: the edge node and the control plane share one upgrade pipeline, not two runbooks. For a bex-style fleet where the management cluster, the workload clusters, and the per-tenant preview environments all hang off the same CAPI control plane, that convergence is the difference between "we upgraded mgmt" and "we upgraded the fleet."

What to Steal, What to Skip, and the One Pre-Flight Check Before You Trust Hands-Off​

Steal the guards, not just the automation. The 11-minute run worked because the pipeline was specific where fleet pipelines are usually vague.

Steal — five checks that belong in every bare-metal fleet pipeline today:

1. Enforce concurrency: 1 for control-plane rollouts, with a policy, not a comment. A Kyverno or ValidatingAdmissionPolicy that rejects concurrency: 0 or missing maxSurge/maxUnavailable is cheaper than an etcd quorum page.
2. Gate the image name with admission control. The quay.io/kairos/hadron:* prefix gate generalizes to "this MachineDeployment may only reference hetzner-workers-v* templates built by the factory." A typo in a template name should fail in the PR, not on a live node.
3. Verify signatures on the OIDC identity, not just the tag. cosign verify --certificate-identity-regexp against the factory's GitHub Actions workload identity. Tag matching alone will not catch a typosquatted registry.
4. Add the recovery dry-run before merge. Calzi's planned kairos-agent upgrade --recovery against the new image in CI catches an unbootable image without touching a live node. On CAPI, the parallel is a canary Machine in a non-production Cluster that runs the new template before the fleet-wide bump fans out.
5. Require the name bump to trigger the rollout. If your CRD is one-shot (NodeOpUpgrade, ClusterDeployment via k0rdent), make Renovate bump the name and the image atomically, and make CI fail if only one changed. At CAPI layer, the infrastructureTemplate is already immutable — use that immutability as the forcing function.

Skip — two pieces that are Kairos-specific and should not be copied literally:

• Do not assume every fleet should run Kairos tomorrow if it already standardizes on Talos or Flatcar. Talos's API-driven, no-SSH model has a stronger drift guarantee than Kairos's "immutable with an off-ramp," and Flatcar's Ignition-based provisioning has deeper Hetzner bare-metal automation today. Pick one immutable base and commit to its upgrade semantics; do not run three.
• Do not run the management cluster's own upgrades and the workload fleets' upgrades through the same concurrency pool. The management cluster's etcd is the control plane for every other cluster's rollout — it upgrades first, alone, with a wider observability window before the fan-out.

The one pre-flight check: before you mark any fleet pipeline "hands-off," run a chaos experiment borrowed from Cluster API's own machine-remediation tests: cordon a worker, kill its kubelet, let MachineHealthCheck remediate it, and assert the workload reschedules without a human. If nodeStartupTimeout plus reprovision never fires in a test, it will not fire at 3 a.m. when a real Hetzner Robot host stops responding to the provisioning API.

Why This Matters for a PaaS That Owns Its Machines​

A managed PaaS sells you freedom from thinking about node upgrades. A self-hosted PaaS on owned Hetzner hardware sells you the opposite — the freedom to decide when nodes upgrade, at the cost of owning the pipeline that does it correctly. The question is not whether upgrades happen; the 2026 CVE stream guarantees they will. The question is whether the pipeline treats etcd quorum as a correctness property enforced by a controller, or as a hope enforced by a runbook.

Kairos's 11-minute pipeline treats it as a property. A/B partitions make rollback a bootloader decision, not an incident procedure. An operator sequencing one node at a time makes "reboot storm" impossible by construction. Cosign makes "wrong image" a signature failure, not a bad deploy. ArgoCD makes "who applied it" an audit trail, not a Slack message. None of those pieces is novel alone. Together, with the two bugs fixed and documented, they are a template a Cluster API fleet can copy without copying the homelab scale.

A team that adopted a single-box Docker PaaS for its first app discovers this seam when a second machine has to exist — that whole category (Coolify, Dokploy, CapRover on awesome-paas) manages the box you already have, not a declarative fleet. A Cluster-API-based platform manages the fleet as Machine objects precisely so that an upgrade is not "SSH into each box" but "bump a template and let controllers reconcile." Kairos plugs the gap those controllers still leave on bare metal: what the OS does when the new bits fail to boot. One layer reconciles machines, the other makes a bad machine self-revert. A bare-metal PaaS needs both.

Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. The same declarative fleet that provisioned your nodes can run the upgrade pipeline above: a management cluster on Hetzner via Cluster API, hardened node images verified by cosign, and a GitOps loop any agent can drive through the Render-compatible API. Star the repo on GitHub or deploy your first app today.