Dora NodaIn October 2024, the CNCF did something it had never done before: it adopted an operating system. Flatcar Container Linux — the community fork of CoreOS Container Linux — entered the incubator, the first OS distribution under the foundation's governance. The endorsement that mattered most came from Adobe, whose principal product manager Joseph Sandoval said the company had "proven it out at very large scale" as the host OS for self-managed Kubernetes across its multicloud environment. Very large scale, in Adobe's case, means 450-plus Kubernetes clusters running some 4,000 applications — on the order of 20,000 production Flatcar nodes.
But the logo is not the story. The story is the update model. Flatcar treats "every node updates itself on a schedule the operator sets" as the default rather than a runbook: dual A-B root partitions, background downloads, and reboots driven by Omaha-protocol channels. If you run a Cluster-API-managed fleet on Hetzner, that default inverts who owns your maintenance windows — and this post gives you the concrete configuration to make the inversion safe.
The short answer, up front
Before the mechanism, here is what breaks and what to do about it.
What breaks in your maintenance windows. Today, OS updates on a CAPH-managed fleet are operator-scheduled calendar events: build a new node image, roll the MachineDeployment, watch new Hetzner Cloud servers come up. With Flatcar's auto-updates on, the OS moves on its own channel cadence while the MachineDeployment template sits unchanged — nodes reboot underneath a template that still describes the old version. That drift is by design, but it means your "maintenance window" stops being a calendar event ("roll the workers Thursday at 02:00") and becomes a reboot-coordination policy: which pools follow which channel, and what drains each node before it reboots. Operate the old calendar on top of self-updating nodes and you get the worst of both — surprise reboots during business hours plus a template that no longer describes reality.
What to configure. Each pool below is a CAPH MachineDeployment with its own HCloudMachineTemplate pointing at a Flatcar snapshot (the image-builder Hetzner target supports Flatcar, and CAPH documents the snapshot flow). The channel pin rides the template as Ignition-written config, so pools can never coincide:
| Pool | CAPH object | Channel pin | Reboot coordinator | CAPI handling |
|---|---|---|---|---|
| Control plane | KubeadmControlPlane / control-plane MD | GROUP=lts-<stream> (frozen LTS stream, e.g. the docs' lts-2022 pattern) | Flatcar Linux Update Operator; locksmithd disabled | Freeze template changes during channel windows; etcd quorum means one control-plane node reboots at a time, ever |
| Workers | Workers MachineDeployment | GROUP=stable | Update Operator + PodDisruptionBudgets | Stagger behind control plane; PDBs cap concurrent disruption per workload |
| Canary | Small-replica canary MD | GROUP=beta | Update Operator | Soaks new releases against real workloads; template promotes to workers only after the canary pool is healthy |
Three moves implement the table. First, write the pin at provision time — update-engine reads /etc/flatcar/update.conf for user overwrites, so an Ignition snippet per template like GROUP=stable (or a frozen GROUP=lts-<stream>) pins the pool from first boot. Second, disable locksmithd on every Kubernetes node and run the Flatcar Linux Update Operator instead, so each reboot is preceded by a cordon and drain. Third, graduate from the public update server to your own Nebraska instance once the fleet grows, so each pool becomes a Nebraska group with its own rollout policy and version. The rest of this post is why each of those moves exists.
What "CNCF custody" actually changed
The timeline matters because a node OS is a multi-year bet. Flatcar was accepted to the CNCF on August 2, 2024 at Incubating level, with the public announcement following on October 29. Microsoft's announcement post called out the auto-update and rollback model explicitly, and Nebraska ships as a sub-project of the distribution. Graduation is now tracked in the open — onboarding, OpenSSF requirements, and a security self-assessment.
Why should a fleet operator care about foundation governance for something as low-level as a node image? Because the failure mode of a node OS is abandonment: CoreOS Container Linux itself was discontinued after the Red Hat acquisition, which is precisely why Flatcar exists as a fork. Neutral custody plus a second at-scale adopter — STACKIT runs its managed Kubernetes engine on Flatcar across 20,000-plus nodes, its customers' most popular OS choice — is the closest thing this layer offers to a guarantee that the update server your nodes phone home to will still exist in three years. Custody does not change a single line of the update protocol; it changes whether betting the fleet on that protocol is prudent.
How Omaha-style updates actually work
The mechanism fits in one paragraph. Each Flatcar node carries two OS partitions. update-engine — speaking the Omaha protocol, the same protocol Chrome uses for browser updates — polls for a new release, downloads it in the background to the passive partition, and, following the configured reboot policy, reboots into it. If the new partition fails to boot, the node rolls back to the previous one automatically. All binaries live on a read-only /usr, so there is no package-by-package drift to reconcile: an update replaces the whole OS image or nothing does.
Channels set the cadence. The documented flow is a promotion pipeline, not parallel product lines:
| Channel | What it carries | Who should follow it |
|---|---|---|
| Alpha | Every new major first, through multiple feature and bug-fix iterations | Developers only; never a fleet pool |
| Beta | Roughly every second major Alpha release, hardened against real user workloads; generally considered production-ready | Your canary pool, for early validation and feedback |
| Stable | The bug-fix release of a new major — never a brand-new major — then ongoing bug fixes; each Stable major stays supported until the next ships | Worker pools |
| LTS | Bug fixes only; a new major stream roughly yearly with 18-month support and a 6-month overlap between streams | Control plane, with the stream frozen |
Two details in that table deserve emphasis because they are the ones operators get wrong. First, Stable never receives a brand-new major release — it receives the hardened bug-fix revision — which is why GROUP=stable is a calmer default than the name alone suggests. Second, the public GROUP=lts pointer follows the current LTS stream, so a stream switchover lands a major-version jump on every node following it. The documented recommendation is to freeze the stream at deploy time (GROUP=lts-<stream>) and move streams deliberately once a year. An unfrozen lts pin on your control plane is a deferred surprise, not a conservative choice.
Fleet-wide control comes from Nebraska, the open-source update server. Nodes poll it over Omaha; operators define groups with distinct rollout policies and release versions. The canary-workers-control-plane stagger in the table above is, at scale, three Nebraska groups — one knob per pool instead of SSH loops.
Where auto-reboot still needs a coordinator
Self-updating nodes plus an uncoordinated reboot policy is how you learn about etcd quorum at 3 a.m. Four gotchas, each with its mitigation.
1. Uncoordinated reboots vs. quorum and disruption budgets. update-engine knows when an update is staged; it knows nothing about your etcd quorum or your tenants' PodDisruptionBudgets. Two control-plane nodes rebooting simultaneously can take the API server offline, and a worker reboot without a drain kills pods that a PDB was supposed to protect. Mitigation: every reboot goes through the Update Operator, and every stateful or quorum-sensitive workload gets a PDB before you enable auto-updates — the PDBs are load-bearing, not decorative.
2. locksmithd vs. the Update Operator on Kubernetes. The CoreOS-inherited locksmithd coordinates reboots via locks, but it does not understand Kubernetes: it will not cordon a node or evict its pods first. On any Kubernetes fleet, disable locksmithd and run the Flatcar Linux Update Operator — an agent on each node watches for the reboot-needed signal and an operator drains the node (cordon plus pod eviction) before allowing the reboot. You can observe the whole flow through the node annotation it sets; the project README shows the annotate-and-watch loop for testing that draining actually happens before your first real update does.
3. MachineDeployment rolls racing OS-driven reboots. CAPI thinks in desired state: change the template and it replaces machines. update-engine thinks in channel cadence: reboot when the staged update says so. If both fire at once — an operator rolling the worker template the same week Stable ships a release — nodes churn twice and debugging gets confusing. Mitigation: treat channel releases as inputs to your rollout calendar. Freeze template changes while a staged update is draining through a pool, and stagger pools — canary, then workers, then control plane via Nebraska groups.
Just as important is the mindset shift: a node whose OS version differs from its template's image is expected drift now, not a broken reconcile to "fix" by reimaging.
4. LTS-vs-Stable choice per pool. Workers want freshness with hardening — Stable. The control plane wants minimal movement — a frozen LTS stream, moved deliberately once a year with a tested rollback plan. The canary pool wants Beta, which the docs explicitly position as the early-validation opportunity so issues get fixed before they reach Stable. Getting this backwards (Beta on control plane, frozen LTS everywhere including the canary) either destabilizes quorum or blinds you to the release your workers will meet next month.
The default flips
Step back and the arc is clear: the industry spent a decade moving from pets to cattle to self-maintaining cattle. Flatcar's CNCF incubation — with graduation tracked openly and two independent 20,000-node fleets vouching for the model — says the self-updating node OS has graduated from CoreOS nostalgia to boring infrastructure. The operator's job does not disappear; it moves up a level, from scheduling maintenance windows to writing the channel pins, reboot policies, and disruption budgets that make maintenance windows unnecessary.
For a self-hosted PaaS on owned Hetzner machines, that move is the whole game: unattended nodes are what let a small team run a fleet without a pager rotation organized around patch Tuesdays. Pin the channels per pool, coordinate every reboot through the Update Operator, and let the OS do the chore it was designed to do.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
